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ABSTRACT 
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Educational Assessment (IDEA) form for collecting student ratings of 
instructional processes and outcomes has been administered since the fall of 
the 1998-1999 school year. Results from all implementations of the instrument 
from August 1998 through August 2001 are the basic data of this report. A 
total of 122 institutions of higher education from different parts of the 
United States participated in this program; reports were prepared for 73,722 
classes, of which 29,267 used a short form and 44,455 used the long 
(Diagnostic) form. The report is organized into six parts: (1) basic data, 
including means, standard deviations, norms for types of institutions, and 
intercorrelations of all items; (2) the structure of the ratings; (3) the 
process of adjusting ratings; (4) reliability; (5) validity; and (6) other 
technical questions. Appendix A, published separately, contains sample forms 
and reports. Appendixes B and C contain details on calculating scores and 
regression coefficients and constants. (Contains 26 tables.) (SLD) 
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Introduction 

A revised version of the IDEA form for collecting student ratings of instructional processes and 

outcomes has been administered since the fall term of the 1998-99 school year^. Results from all 
administrations of the device from August 1998, through August 2001, constitute the basic data of this 
report. A total of 122 institutions of higher education participated in the program during this time span; 

[21 

reports were prepared for 73,722 classes , of which 29,267 used the Short Form and 44,455 used the 
Diagnostic (long) Form. 

No claim is made that participants are representative of American higher education. However, they are 
relatively diverse, both geographically and in mission. Table 1 shows information about the highest 
degree offered by participating institutions as well as their geographic location. 



Table 1 

Number of Institutions Included in Research 



Location 


Highest Degree Offered 


Associate 


Baccalau- 

reate 


Master’s 


Doctoral 


Other 


Total 


Southeast 


4 


2 


4 


2 


3 


15 


East/Northeast 


7 


5 


9 


5 


0 


26 


Midwest 


8 


5 


17 


10 


8 


48 


Southwest 


5 


3 


5 


4 


1 


18 


Rockies/West 


4 


5 


2 


4 


0 


15 


Total 


28 


20 


37 


25 


12 


122 



Fifty-five institutions were publicly supported, 44 were private not-for-profit, of which many were 
church related, and 23 were private for-profit. Enrollment varied widely from under 500 (1 1 
institutions) to over 20,000 (9 institutions). The two most common size categories were 1000-2499 (28 
institutions) and 5000-9999 (29 institutions). 

In terms of classes processed, 22 percent were from two-year institutions, 14 percent from those whose 
highest degree offered was the bachelor’s, 28 percent from Master’s degree institutions, 23 percent from 
doctoral institutions, and 13 percent from other types of institutions. 

This report is organized into six parts. 

I. Basic Data (including means, standard deviations, norms for types of institution, and inter- 

correlations of all items) 

II. The Structure of the Ratings 

in. The Process of Adjusting Ratings 

IV. Reliability 

V. Validity 

VI. Other Technical Questions 

Section I. Basic Data 

This section presents item means, standard deviations, and inter-correlations as well as percentile ranks 
for all institutions and for each of four types of institutions (defined by highest degree offered). The 
data are based on the 44,455 classes that employed the Diagnostic Form in the time period from August 
1998, through August 2001. 

3 
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Table 2 describes faculty ratings of the importance of the 12 learning objectives as reported on the 
Faculty Information Form (FIF). A 3-point rating scale was used for these 12 items: “l=Of no more 
than minor importance;” “2=Important;” and “3=Essential.” The table shows the number of classes for 
which a given objective was identified as “important” or “essential,” the mean and standard deviation, 
and the percent of classes where the objective was identified as “essential” or “important.” 

Table 2 



Faculty Ratings of the Importance of Twel 


ve Learning Objectives 


Learning Objective 


N 

(Important & 
Essential) 


% 

Impor- 

tant^ 


% Essen- 
tial “ 


Mean** 


s.d. 


1. Gaining factual knowledge (trends, etc.) 


31,991 


32 


46 


2.24 


.79 


2. Learning fundamental principles, 
generalizations, or theories 


30,398 


34 


41 


2.16 


.80 


3. Learning to apply course material (to 
improve thinking, problem solving, and 
decisions) 


30,442 


40 


35 


2.10 


.77 


4. Developing skills, competencies, and 
points of view needed by professionals 


21,568 


30 


25 


1.80 


.81 


5. Acquiring skills in working as a team 
member 


12,088 


24 


8 


1.39 


.63 


6. Developing creative capacities— writing, 
art, etc 


9,290 


15 


10 


1.34 


.65 


7. Gaining a broad understanding, 
appreciation of intellectual/cultural 
activity (music, science, etc.) 


10,256 


17 


10 


1.37 


.66 


8. Developing skill in expressing oneself 
orally or in writing. 


18,174 


26 


20 


1.67 


.79 


9. Learning how to find and use resources 


15,656 


31 


10 


1.51 


.67 


10. Developing a clearer understanding of, and 
commitment to, personal values. 


8,715 


17 


6 


1.30 


.58 


11. Learning to analyze and critically judge 
ideas 


18,909 


29 


20 


1.68 


.78 


12. Acquiring an interest in learning more 


15,616 


30 


11 


1.52 


.68 



“Percentages based on all classes employing the Diagnostic Form. Percentages will not equal 100 because the percentage 
indicting the objective was “Of minor or no importance” are not reported. 

3-point rating scale was used: l=Of no more than minor importance, 2=Important, 3=Essential. 



A review of Table 2 provides an indication of the instructional priorities of those participating in the 
IDEA program. The first four objectives are stressed most frequently; these represent the acquisition 
and application of basic cognitive background, often as a part of professional preparation. Academic 
skills (8. communication; 11. critical analysis) were also stressed frequently, but not as often as the first 
four objectives. Next in importance were the two “life-long learning” objectives (9. finding and using 
resources; 12. interest in learning more). The objectives that were stressed least were those concerned 
with values development (item 10), creative capacities (item 6), and a broad liberal education (item 7). 
American higher education is often portrayed as pragmatic and utilitarian; these results are consistent 
with that stereotype. 

Table 3 gives the mean, standard deviation, and number of classes for the 47 individual items rated by 
students. A 5-point rating scale was used throughout, with “1” representing the lowest rating (least 
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frequent, least characteristic, least satisfactory) and “5” the highest rating. 

In addition, two “overall effectiveness” measures were included — PRO (Progress on Relevant 
Objectives) and PRO^^j. PRO was derived by combining the faculty member’s ratings of “Importance” 

of a given objective with the average student rating of “Progress” on that objective. Because the average 
student rating of progress is different for each of the 12 learning objectives, these averages were first 
expressed as T Scores, a mathematical way of converting all averages to 50 and all standard deviations 

to 10^. These T Scores were then weighted by the faculty member’s rating of the importance 
(relevance) of each objective. For objectives rated as “Essential,” the T Score was multiplied by 2 
before being added to the T Score for objectives chosen as “Important;” objectives rated as “Of no more 
than minor importance” were ignored. The PRO measure was derived by dividing the sum of the 
weighted T Scores by the sum of the weights. The PRO^^j measure adjusts PRO by taking into account 

factors which influence student ratings but which are beyond the control of the instructor. The 
adjustment process is described in Section III of this report. 

For the student ratings shown in Table 3, it should be noted that, although “3” was the midpoint of the 
rating scale, all ratings averaged above “3” and 13 of them averaged above “4.” While these relatively 
high ratings probably reflect a generally high quality of instruction being provided at participating 
institutions, they are also due in part to a tendency for students to be “lenient” in their ratings. This is 
revealed most clearly in those items where students are asked to compare the class with others they have 
taken (Items 33-35), where averages were 3.20, 3.42, and 3.42, respectively — well above the average 
which would be expected if leniency were not an issue. 



5 
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Table 3 

Student Ratings of Individual Items on the IDEA Diagnostic Form 



Student Ratinj^s ofTeachinf^ Methods 


N 


Mean 


s.d. 


1. Displayed a personal interest in students and their learning. 


44,451 


4.34 


.50 


2. Found ways to help students answer their own questions. 


44,448 


4.10 


.52 


3. Scheduled course work in ways which encouraged students to stay up-to-date in 
their work. 


44,447 


4.20 


.48 


4. Demonstrated the importance and significance of the subject. 


44,447 


4.32 


.45 


5. Formed “teams” or “discussion groups” to facilitate learning. 


44,446 


3.52 


1.03 


6. Made it clear how each topic fit into the course. 


44,444 


4.20 


.51 


7. Explained criticisms of students academic performance. 


44,445 


3.78 


.57 


8. Stimulated students to intellectual effort beyond that required by most courses. 


44,443 


3.86 


.57 


9. Encouraged students to use multiple resources to improve understanding. 


44,444 


3.78 


.70 


10. Explained course material clearly and concisely. 


44,446 


4.13 


.61 


11. Related course material to real life situations. 


44,444 


4.22 


.58 


12. Gave tests, projects, etc. that covered the most important points of the course. 


44,440 


4.28 


.49 


13. Introduced stimulating ideas about the subject. 


44,443 


4.03 


.58 


14. Involved students in “hands on” projects (research, etc.). 


44,443 


3.76 


.80 


15. Inspired students to set and achieve goals which really challenged them. 


44,446 


3.76 


.62 


16. Asked students to share ideas and experiences with others with different 
backgrounds and viewpoints. 


44,445 


3.69 


.79 


17. Provided timely and frequent feedback on tests, projects, etc. 


44,443 


4.11 


.59 


18. Asked students to help each other understand ideas, concepts. 


44,444 


3.79 


.64 


19. Gave projects, tests, etc. that required original thinking. 


44,445 


3.92 


.65 


20. Encouraged student-faculty interaction outside of class. 


44,446 


3.90 


.63 


44. Used a variety of methods to evaluate student progress. 


44,442 


3.83 


.60 


45. Expected students to take their share of responsibility for learning. 


44,442 


4.30 


.33 


46. Had high achievement standards in this class. 


44,442 


4.13 


.41 


47. Used educational technology to promote learning. 


44,442 


3.63 


.77 


Student Ratings of Progress 


21. Gaining factual knowledge (terminology, etc.) 


44,443 


3.94 


.52 


22. Learning fundamental principles, generalizations, or theories 


44,442 


3.89 


.51 


23. Learning to apply course material (to improve thinking, problem solving, and 
decisions) 


44,440 


3.95 


.52 


24. Developing skills, competencies, and points of view needed by professionals in the 
field most closely related to this course 


44,441 


3.91 


.54 


25. Acquiring skills in working with others as a team member 


44,437 


3.45 


.82 


26. Developing creative capacities (writing, inventing, etc.) 


44,438 


3.37 


.79 


27. Gaining a broader understanding and appreciation of intellectual/cultural activity 
(music, science, literature, etc.) 


44,440 


3.32 


.74 


28. Developing skill in expressing oneself orally or in writing 


44,439 


3.41 


.80 


29. Learning how to find and use resources for answering questions or solving 
problems 


44,435 


3.58 


.60 


30. Developing a clearer understanding of, and commitment to, personal values 


44,434 


3.44 


.69 


31. Learning to analyze and critically evaluate ideas, etc. 


44,436 


3.67 


.63 


32. Acquiring an interest in learning more 


44,437 


3.74 


.56 


Ratin^^s of Course Characteristics 


33. Amount of reading 


44,447 


3.20 


.74 


34. Amount of work in other (non-reading) assignments 


44,445 


3.42 


.59 


35. Difficultv of subject matter 


44,445 


3.42 


.58 


Self-Ratings 


36. 1 had a strong desire to take this course. 


44,44/ 


5.66 


1 ...» . 
.0/ 


37. 1 worked harder on this course than on most I have taken. 


44,448 


3.57 


.56 


38. 1 really wanted to take a course from this instructor. 


44,447 


3.40 


.67 


39. 1 really wanted to take this course regardless of who taught it. 


44,447 


3.33 


.56 


43. As a rule, I put forth more effort than other students on my academic work. 


44,443 


3.64 


.31 


Global Ratings of Outcomes 
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40. As a result of taking this course, I have more positive feelings toward this field of 
study. 


44,447 


3.86 


.60 


41. Overall, I rate this instructor an excellent teacher. 


44,447 


4.18 


.64 


42. Overall, I rate this course as excellent. 


44,447 


3.92 


.61 


Progress on Relevant Objectives (PRO)^ 


42,785 


50.9 


8.7 


PRO- Adjusted 


42,344 


51.0 


8.5 



^PRO ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10. All other ratings 
were made on a 5-point scale where 1 is low and 5 is high. 



Inter-correlations for all items included in Tables 2 and 3 are provided in Tables 4, 5, and 6. Refer to 
Tables 2 and 3 for item descriptions. 

The correlations shown in these tables may seem overwhelming. Aside from their value as basic 
information, they can help the reader gain a deeper understanding of individual ratings. For example, 
there may be interest in understanding factors that relate to how hard students work in a class (Item 37: 
“I worked harder on this course than on most courses I have taken”). As shown in Table 6, although a 
substantial number of items were significantly correlated with responses to this item, the highest 
correlations were with items related to the instructor’s course management and/or expectations. Thus, 
means on this item correlated .68 with the amount of other (non-reading) work assigned in the course 
(Item 34), .67 with the difficulty of the course (Item 35), .66 with the instructor’s achievement standards 
(Item 46), and .54 with the instructor’s tendency to hold students responsible for their own learning 
(Item 45). Similarly, the perceived difficulty of a course (Item 35) was largely a function of the 
magnitude of assignments given (reading. Item 33; other. Item 34) as well as the instructor’s 
achievement standards (Item 46) and success in stimulating student effort (Item 8). Detailed analyses 
such as these can result in new insights regarding teaching, learning, and the IDEA system. 



Table 4 

Inter-Correlations of IDEA Faculty Information Form 



Item 


FRl 


FR2 


FR3 


FR4 


FR5 


FR6 


FR7 


FR8 


FR9 


FRIO 


FRll 


FR12 


FRl 


1.00 
























FR2 


.42 


1.00 






















FR3 


.13 


.28 


1.00 




















FR4 


.13 


.10 


.30 


1.00 


















FR5 


-.03 


.04 


.27 


.26 


1.00 
















FR6 


-.11 


-.04 


.13 


.21 


.29 


1.00 














FR7 


-.04 


-.01 


-.03 


-.04 


.12 


.33 


1.00 












FR8 


-.22 


-.14 


.06 


.01 


.31 


.34 


.24 


1.00 










FR9 


.07 


.10 


.32 


.25 


.34 


.28 


.17 


.38 


1.00 








FRIO 


-.00 


.08 


.21 


.10 


.29 


.22 


.26 


.26 


.32 


1.00 






FRll 


-.11 


.07 


.23 


.00 


.22 


.24 


.27 


.46 


.41 


.38 


1.00 




FR12 


.13 


.20 


.33 


.22 


.34 


.30 


.30 


.32 


.52 


.45 


.50 


1.00 



See Table 2 for item descriptions. 
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Table 5 

Inter-Correlations of IDEA Faculty Information Form (FR) 



Item 


FRl 


FR 2 


FR 3 


FR 4 


FR 5 


FR 6 


FR 7 


FR 8 


FR 9 


FRIO 


FRll 


FR 12 


SRI 


-.07 


-.06 


.00 


.05 


.04 


.05 


.00 


.04 


.01 


.07 


.00 


.03 


SR 2 


-.08 


-.06 


.03 


.05 


.04 


.04 


-.01 


.04 


.01 


.07 


.02 


.04 


SR 3 


-.03 


-.05 


.02 


.04 


.00 


.02 


-.03 


.03 


-.01 


.00 


-.03 


-.01 


SR 4 


.02 


-.01 


.01 


.06 


.00 


-.01 


-.03 


-.02 


-.02 


.09 


-.02 


.02 


SR 5 


-.24 


-.18 


.06 


.06 


.36 


.08 


-.02 


.23 


.08 


.12 


.10 


.04 


SR 6 


.01 


-.03 


-.01 


.03 


.02 


-.02 


-.01 


-.01 


-.04 


.07 


-.02 


.00 


SR 7 


-.15 


-.12 


.01 


.09 


.09 


.16 


.02 


.14 


.04 


.06 


.05 


.03 


SR 8 


-.05 


-.03 


.03 


.05 


.03 


.04 


.00 


.05 


.02 


.04 


.06 


.03 


SR 9 


-.14 


-.14 


.02 


.07 


.12 


.10 


-.01 


.21 


.22 


.06 


.12 


.06 


SRIO 


.00 


-.03 


-.03 


-.02 


-.03 


.00 


.01 


.02 


-.03 


.04 


-.01 


.00 


SRll 


.02 


.02 


.07 


.07 


.06 


-.07 


-.10 


-.02 


.00 


.14 


.02 


.03 


SR 12 


.13 


.07 


.02 


.01 


-.06 


-.10 


-.06 


-.11 


-.06 


-.02 


-.09 


-.03 


SR 13 


-.04 


-.05 


-.02 


.02 


.03 


.05 


.07 


.04 


.00 


.13 


.06 


.06 


SR 14 


-.12 


-.13 


.10 


.23 


.25 


.13 


-.08 


.08 


.15 


.07 


.00 


.04 


SR 15 


-.12 


-.10 


.06 


.15 


.13 


.14 


-.03 


.08 


.08 


.09 


.02 


.05 


SR 16 


-.22 


-.17 


.00 


.03 


.17 


.12 


.06 


.24 


.09 


.23 


.19 


.12 


SR 17 


.01 


.00 


.00 


.01 


-.02 


-.02 


-.03 


.02 


-.03 


.00 


-.02 


-.01 


SR 18 


-.17 


-.13 


.05 


.10 


.20 


.09 


-.02 


.12 


.05 


.10 


.05 


.05 


SR 19 


-.24 


-.18 


.03 


.09 


.14 


.24 


.07 


.26 


.11 


.10 


.15 


.07 


SR 20 


.06 


-.05 


.01 


.03 


.04 


-.02 


-.06 


.03 


.02 


.00 


.01 


-.01 


SR 21 


.21 


.11 


.04 


.12 


-.05 


-.09 


-.10 


-.17 


-.05 


-.05 


-.11 


-.02 


SR 22 


.14 


.17 


.09 


-.11 


-.02 


-.07 


-.13 


-.17 


-.06 


-.01 


-.07 


.00 


SR 23 


-.04 


-.01 


.14 


.19 


.07 


.03 


-.16 


-.03 


.02 


.04 


-.04 


.01 


SR 24 


.00 


-.03 


.08 


.26 


.08 


.07 


-.14 


-.04 


.02 


-.00 


-.08 


.00 


SR 25 


-.18 


-.14 


.10 


.15 


39 


.08 


-.07 


.14 


.09 


.08 


.02 


.04 


SR 26 


-.32 


-.27 


-.04 


.10 


.17 


.37 


.17 


.35 


.12 


.11 


.16 


.09 


SR 27 


-.18 


-.18 


-.11 


-.02 


.08 


.25 


.33 


.22 


.05 


.14 


.14 


.11 


SR 28 


-.32 


-.26 


-.04 


.01 


.17 


.19 


.12 


.46 


.13 


.16 


.24 


.09 


SR 29 


-.10 


-.10 


.08 


.12 


.12 


.05 


-.09 


.16 


.21 


.02 


.08 


.05 


SR 30 


-.16 


-.11 


.03 


.05 


.13 


.08 


.02 


.15 


.08 


.28 


.15 


.11 


SR 31 


-.21 


-.12 


.02 


-.02 


.08 


.08 


.03 


.23 


.07 


.16 


.27 


.08 


SR 32 


-.09 


-.06 


.05 


.10 


.08 


.07 


-.02 


.06 


.06 


.11 


.08 


.09 


SR 33 


.01 


.01 


-.04 


-.13 


-.05 


-.18 


.08 


.13 


.00 


.06 


.21 


.03 


SR 34 


-.06 


-.05 


.12 


.19 


.08 


.12 


-.12 


.06 


.07 


-.13 


-.06 


-.05 


SR 35 


.16 


.17 


.05 


.02 


-.12 


-.11 


-.08 


-.16 


-.08 


-.18 


-.05 


-.07 


SR 36 


.08 


.03 


.03 


.26 


.07 


.11 


-.04 


-.11 


-.02 


.05 


-.10 


.05 


SR 37 


.04 


.03 


.07 


.16 


.01 


.06 


-.10 


-.02 


.00 


-.10 


-.04 


-.03 


SR 38 


-.01 


-.03 


.01 


.13 


.04 


.04 


-.04 


-.06 


-.03 


.02 


-.07 


-.01 


SR 39 


.08 


.04 


.06 


.25 


.09 


.10 


-.05 


-.09 


.01 


.03 


-.10 


.05 


SR 40 


.04 


-.01 


.02 


.18 


.05 


.07 


-.02 


-.06 


-.02 


.08 


-.06 


.04 


SR 41 


-.03 


-.05 


-.03 


.00 


-.01 


.02 


.01 


.02 


-.03 


.04 


.00 


.00 


SR 42 


.00 


-.03 


-.01 


.11 


.03 


.08 


.00 


-.01 


-.03 


.07 


-.04 


.03 


O'n A n 


nn 

.\j\j 




m 

.\j / 


1 n 

• X / 


AO 

■ v/v 


05 


-.03 


c > 


.02 


.01 


-.04 


-Q1 


SR 44 


-.12 


-.12 


.08 


.15 


.16 


.09 


-.03 


.12 


.07 


.05 


-.01 


.02 


SR 45 


-.04 


-.06 


.01 


.10 


.04 


.03 


-.03 


.01 


-.01 


.01 


-.01 


.00 


SR 46 


-.03 


-.05 


.02 


.10 


.02 


.05 


-.04 


.04 


-.01 


-.01 


.00 


p 

r 


SR 47 


.00 


-.07 


.07 


.14 


.09 


-.01 


-.10 


.00 


.14 


-.07 


-.05 


-.01 



Bold numbers are correlations between student ( SR 21 - SR 32 ) and faculty ratings ( FR 1 - FR 12 ) of the twelve learning 
objectives . 
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See Tables 2 and 3 for item descriptions. 
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Table 6 



Inter-Corre 


ations of IDEA Student Ratings (SR) - Diagnostic 


Form 




SRI 


SR2 


SR3 


SR4 


SR5 


SR6 


SR7 


SR8 


SR9 


SRIO 


SRll 


SR12 


SR13 


SR14 


SRI 5 


SR16 


SI 
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Item 




































SRI 


1.0 


































SR2 


.88 


1.0 
































SR3 


.72 


.76 


1.0 






























SR4 


.79 


.81 


.73 


1.0 




























SR5 


.41 


.44 


.36 


.33 


1.0 


























SR6 


.78 


.81 


.74 


.90 


.39 


1.0 
























SR7 


.76 


.79 


.69 


.71 


.48 


.74 


1.0 






















SR8 


.73 


.80 


.70 


.76 


.40 


.75 


.76 


1.0 




















SR9 


.54 


.56 


.49 


.54 


-.48 


.53 


.60 


.61 


1.0 


















SRIO 


.77 


.81 


.76 


.83 


.27 


.86 


.71 


.69 


.48 


1.0 
















SRll 


.64 


.65 


.55 


.78 


.36 


.77 


.57 


.60 


.49 


.67 


1.0 














SR12 


.64 


.67 


.73 


.72 


.19 


.74 


.57 


.62 


.38 


.75 


.59 


1.0 












SR13 


.78 


.82 


.69 


.86 


.40 


.86 


.74 


.79 


.59 


.81 


.79 


.68 


1.0 










SR14 


.52 


.54 


.47 


.51 


.64 


.52 


.58 


.52 


.68 


.41 


.55 


.34 


.58 


1.0 








SR15 


.77 


.81 


.69 


.75 


.51 


.74 


.82 


.84 


.67 


.69 


.62 


.56 


.79 


.70 


1.0 






SR16 


.63 


.65 


.49 


.59 


.64 


.61 


.66 


.60 


.65 


.53 


.64 


.37 


.72 


.64 


.70 


1.0 




SR17 


.66 


.67 


.71 


.64 


.26 


.66 


.65 


.61 


.41 


.70 


.52 


.68 


.62 


.35 


.59 


.45 


1 


SR18 


.71 


.76 


.61 


.61 


.72 


.64 


.73 


.68 


.58 


.57 


.54 


.48 


.67 


.65 


.77 


.75 




SR19 


.61 


.65 


.59 


.58 


.56 


.59 


.70 


.66 


.68 


.54 


.50 


.45 


.68 


.69 


.74 


.74 




SR20 


.74 


.70 


.61 


.62 


.38 


.63 


.67 


.68 


.55 


.59 


.53 


.54 


.64 


.47 


.69 


.53 




SR21 


.60 


.66 


.62 


.72 


.18 


.73 


.57 


.72 


.42 


.68 


.59 


.69 


.68 


.40 


.63 


.36 




SR22 


.61 


.68 


.62 


.72 


.22 


.71 


.59 


.73 


.41 


.67 


.60 


.67 


.69 


.41 


.65 


.41 




SR23 


.70 


.77 


.68 


.76 


.40 


.74 


.70 


.76 


.53 


.69 


.68 


.63 


.74 


.60 


.78 


.57 




SR24 


.67 


.72 


.64 


.74 


.37 


.73 


.70 


.73 


.53 


.67 


.64 


.60 


.71 


.61 


.78 


.54 




SR25 


.46 


.51 


.41 


.41 


.86 


.44 


.53 


.48 


.51 


.34 


.42 


.27 


.46 


.71 


.61 


.62 




SR26 


.50 


.54 


.46 


.44 


.54 


.46 


.66 


.54 


.61 


.44 


.35 


.27 


.57 


.61 


.67 


.69 




SR27 


.52 


.57 


.46 


.51 


.40 


.53 


.62 


.59 


.51 


.52 


.37 


.36 


.66 


.44 


.62 


.64 




SR28 


.50 


.54 


.45 


.47 


.58 


.49 


.63 


.57 


.66 


.45 


.43 


.29 


.59 


.56 


.63 


.76 




SR29 


.57 


.63 


.56 


.56 


.46 


.56 


.63 


.68 


.82 


.53 


.49 


.46 


.60 


.65 


.72 


.60 




SR30 


.61 


.66 


.52 


.64 


.50 


.64 


.66 


.65 


.62 


.59 


.63 


.43 


.73 


.57 


.73 


.80 




SR31 


.57 


.65 


.52 


.60 


.48 


.61 


.66 


.72 


.63 


.56 


.56 


.42 


.70 


.51 


.68 


.75 




SR32 


.72 


.80 


.65 


.72 


.44 


.71 


.73 


.81 


.61 


.68 


.61 


.57 


.79 


.56 


.81 


.69 




SR33 


.01 


.05 


.04 


.10 


.10 


.10 


.03 


.24 


.19 


.02 


.13 


.05 


.15 


.00 


.06 


.19 


.1 


SR34 


.11 


.15 


.24 


.07 


.20 


.03 


.21 


.33 


.27 


-.01 


-.06 


.09 


.02 


.27 


.32 


.05 




SR35 


-.05 


.01 


.02 


.01 


-.14 


-.03 


-.01 


.30 


-.03 


-.10 


-.09 


.07 


-.03 


-.13 


.06 


-.22 


.1 


SR36 


.39 


.41 


.32 


.46 


.17 


.45 


.39 


.42 


.27 


.37 


.41 


.32 


.50 


.38 


.46 


.34 




SR37 


.24 


.30 


.31 


.30 


.13 


.25 


.32 


.56 


.28 


.18 


.14 


.24 


.27 


.22 


.45 


.13 




SR38 


.67 


.69 


.56 


.66 


.31 


.67 


.65 


.67 


.46 


.64 


.57 


.53 


.70 


.48 


.69 


.50 




SR39 


.22 


.23 


.19 


.28 


.12 


.27 


.25 


.24 


.16 


.21 


.24 


.18 


.31 


.27 


.30 


.21 




SR40 


.68 


.70 


.61 


.77 


.30 


.76 


.64 


.66 


.47 


.70 


.67 


.60 


.79 


.53 


.70 


.57 




SR41 


.85 


.86 


.76 


.83 


.32 


.84 


.74 


.75 


.50 


.90 


.66 


.73 


.83 


.45 


.74 


.56 




SR42 


.73 


.76 


.68 


.80 


.31 


.80 


.69 


.72 


.48 


.79 


.66 


.66 


.82 


.50 


.74 


.57 


.1 


SR43 


.19 


.23 


.20 


.24 


.21 


.24 


.29 


.33 


.24 


.13 


.21 


.14 


.25 


.30 


.36 


.22 




SR44 


.61 


.62 


.64 


.56 


.56 


.56 


.63 


.58 


.59 


.50 


.47 


.49 


.57 


.69 


.68 


.56 




SR45 


.56 


.59 


.56 


.59 


.31 


.56 


.55 


.67 


.44 


.48 


.43 


.48 


.56 


.41 


.62 


.40 




SR.46 


.54 


.58 


.56 


.60 


29 


.56 


-58 


.74 


.46 


.49 


.41 


.46 


.56 


.40 


.68 


.39 




SR47 


.33 


.35 


.36 


.32 


.30 


.32 


.34 


.36 


.55 


.28 


.32 


.30 


.33 


.49 


.41 


.32 





Table 6 (continued) 

Inter-Correlations of IDEA Student Ratings (SR) - Diagnostic Form 





SR25 


SR26 


SR27 


SR28 


SR29 


SR30 


SR31 


SR32 


SR33 


SR34 


SR35 


SR36 


SR37 


SR38 


SR39 
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SR25 


1.0 






























SR26 


.58 


1.0 




























SR27 


.46 


.79 


1.0 


























SR38 


.59 


.84 


.71 


1.0 
























SR29 


.59 


.62 


.53 


.68 


1.0 






















SR30 


.60 


.68 


.69 


.74 


.68 


1.0 




















SR31 


.53 


.67 


.64 


.78 


.71 


.80 


1.0 


















SR32 


.57 


.63 


.65 


.65 


.76 


.79 


.81 


1.0 
















SR33 


.06 


.06 


.15 


.26 


.19 


.20 


.33 


.17 


1.0 














SR34 


.26 


.26 


.09 


.18 


.36 


.09 


.17 


.24 


.17 


1.0 












SR35 


-.09 


-.17 


-.07 


-.14 


.08 


-.12 


.06 


.11 


.40 


.49 


1.0 










SR36 


.30 


.33 


.35 


.26 


.33 


.41 


.32 


.50 


.04 


.12 


.06 


1.0 








SR37 


.25 


.25 


.23 


.23 


.41 


.25 


.34 


.45 


.33 


.68 


.67 


.41 


1.0 






SR38 


.43 


.44 


.46 


.43 


.54 


.56 


.53 


.67 


.05 


.15 


.11 


.58 


.38 


1.0 




SR39 


.24 


.24 


.24 


.16 


.23 


.28 


.18 


.34 


.04 


.13 


.05 


.79 


.34 


.27 


1.0 


SR40 


.43 


.49 


.54 


.47 


.54 


.64 


.57 


.74 


.07 


.09 


-.02 


.74 


.37 


.70 


.55 


SR41 


.40 


.47 


.54 


.47 


.56 


.60 


.59 


.73 


.02 


.06 


-.03 


.41 


.25 


.73 


.22 


SR42 


.43 


.52 


.57 


.50 


.56 


.65 


.60 


.76 


.04 


.09 


-.02 


.69 


.37 


.72 


.50 


SR43 


.28 


.26 


.25 


.25 


.31 


.28 


.26 


.32 


.14 


.30 


.24 


.33 


.43 


.35 


.29 


SR44 


.61 


.59 


.47 


.57 


.62 


.54 


.51 


.60 


.03 


.40 


-.07 


.34 


.30 


.50 


.24 


SR45 


.38 


.38 


.40 


.40 


.49 


.45 


.48 


.60 


.21 


.37 


.27 


.42 


.54 


.51 


.27 


SR46 


.37 


.41 


.42 


.42 


.52 


.46 


.52 


.61 


.25 


.47 


.39 


.38 


.66 


.52 


.25 


SR47 


.37 


.32 


.23 


.28 


.53 


.30 


.29 


.37 


.09 


.28 


.04 


.22 


.22 


.30 


.17 



See Table 3 for item descriptions. 
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Of special interest is the relationship between ratings of teaching methods and instructional outcomes. 
Are some teaching approaches more closely associated with progress of a given type than others? Do 
the most effective methods differ depending on instructor objectives? Answers to these questions are 
highly relevant to the IDEA system’s goal of facilitating instructional improvement. 

Although a review of relevant correlations in Tables 4, 5, and 6 provides a direct approach to this 
problem, it is commonly assumed that answers may depend, in part, on class size. Therefore, 
correlations between instructional methods and student ratings of progress were computed separately for 
four class sizes — small (10-14), medium (15-34), large (35-49), and very large (50+). Table 7 shows the 
“methods” items, which were most closely related to progress ratings on each objective for each of these 
four class sizes. Typically, seven to ten methods were identified as “most” closely related to progress 
ratings. 

Although there was some overlap between the lists of “most relevant” items (especially between the first 
two objectives), the pattern of items tended to be distinctive for each objective. Differences among class 
sizes were not dramatic, but were large enough to merit a separate listing of “most relevant items” for 
each size group. 
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Table 7 





Obj. 21. Gaining 
Factual Knowledge 




Obj. 22. Principles and 
Theories 




Obj. 23. Applications 




S 


M 


L 


VL 


S 


M 


L 


VL 


S 


M 


L 


VL 


1. Displayed psnl 
interest in Ss 


















.69 


.71 






2. Helped Ss answ 
own Qs 


.65 


.69 


.69 


.66 


.68 


.71 


.73 


.75 


.75 


.78 


.77 


.75 


3. Scheduled work 
helpfully 


.64 
















.69 








4. Demonstrated imp 
of subject 


.70 


.73 


.74 


.73 


.69 


.72 


.72 


.73 


.76 


.79 


.78 


.76 


5. Formed teams, 
discussion 


























6. Made clear how 
topics fit 


.71 


.74 


.75 


.72 


.70 


.73 


.73 


.73 


.75 


.78 


.76 


.75 


7. Explained 
criticisms 




















.71 


.73 


.73 


8. Stimulated 
intellectual effort 


.73 


.76 


.78 


.78 


.74 


.77 


.78 


.79 


.73 


.78 


.79 


.78 


9. Encrgd multiple 
resources 


























10. Explained clearly 


.67 


.70 


.72 


.70 


.67 


.69 


.70 


.71 


.69 


.71 


.70 




11. Related to real 
life 


.64 
















.69 


.70 




.68 


12. Tests cover imprt. 
points 


.68 


.69 


.70 


.69 


.65 


.68 


.68 


.74 










13. Introduce 
stimulating ideas 


.67 


.71 


.70 


.68 


.67 


.71 


.69 


.70 


.74 


.77 


.74 


.71 


14. Involved Ss in 
“hands on” 


























15. Inspired to set 
high goals 


.65 


.66 


.69 


.65 


.66 


.68 


.69 


.71 


.76 


.79 


.80 


.80 


16. Asked to share 
experiences 


























17. Provided timely 
feedback 


























18. Asked Ss to help 
each other 


























19. Creative 
assessments 


























20. Enrgd out class 
S/F contact 




























Obj. 24. Prof. Skills, 
Viewpoints 




Obj. 25. Team Skills 




Obj. 26. Creative 
Capacities 




S 


M 


L 


VL 


S 


M 


L 


VL 


S 


M 


L 


VL 


1. Displayed psnl 
interest in Ss 


.67 


.70 














.54 








2. Helped Ss answ 
own Qs 


.72 


.76 


.75 


.74 


.53 


.52 




.57 


.53 


.57 


.63 


.60 


3. Scheduled work 
helpfully 


























4. Demonstrated imp 
of subject 


nc 


79 


79 


n'x 


















5. Formed teams, 
discussion 










.75 


.77 


.77 


.70 








.62 


6. Made clear how 
topics fit 


.75 


.79 


.78 


.71 


.52 
















7. Explained 
criticisms 


.68 


.72 


.73 


.73 


.54 




.54 


.62 


.63 


.67 


.73 


.69 



er|c 



14 

http;//www.idea.ksu.edu/users/tchrptl2/TECHNICAL%20REPORT%2012.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 16 of 33 



8. Stimulated 
intellectual effort 


.71 


.76 


.78 


.77 






.52 


.53 






.53 


.56 






9. Encrgd multiple 
resources 






























10. Explained clearly 


.69 


.71 


.70 
























1 1 . Related to real 
life 




.69 


























12. Tests cover imprt. 
points 






























13. Introduce 
stimulating ideas 


.73 


.77 


.75 


.69 














.57 


.58 


.65 


.60 


14. Involved Ss in 
'‘hands on** 












.67 


.67 


.68 


.72 




.52 




.63 


.72 


15. Inspired to set 
high goals 


.76 


.78 


.80 


.79 




.60 


.59 


.61 


.70 




.68 


.66 


.73 


.78 


16. Asked to share 
experiences 














.53 








.53 


.59 


.65 


.73 


17. Provided timely 
feedback 






























18. Asked Ss to help 
each other 








.68 




.63 


.67 


.65 


.70 




.55 


.57 


.69 


.79 


19. Creative 
assessments 














.53 


.56 


.63 




.74 


.78 


.73 


.64 


20. Enrgd out class 
S/F contact 































S=small (10-14), M=medium (15-34), L=large (35-49), VL=very large (50-t-) 

Only the most highly correlated items are shown. 

Note: Analyses reported in Table 7 used a more restricted data set. Classes with response rates less than 75% or not reporting 
the number enrolled were also excluded. 
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Table 7 (continued) 

Relationship of Teaching Methods to Learning Ob.iectives (Correlations) 





Obj. 27. Broad Liberal 
Education 




Obj. 28. Communi- 
cation Skills 




29. Find, Use Resources 




s 


M 


L 


VL 


s 


M 


L 


VL 


s 


M 


L 


VL 


1. Displayed psnl 
interest in Ss 


.50 










.55 














2. Helped Ss answ own 
Os 


.51 


.59 


.56 


.52 


.56 


.58 


.58 


.59 


.64 


.65 


.66 


.64 


3. Scheduled work 
helpfully 


























4. Demonstrated imp of 
subject 






.57 


.52 


















5. Formed teams, 
discussion 


























6. Made clear how 
topics fit 


.50 


.58 


.58 


.54 


















7. Explained criticisms 


.56 


.62 


.62 


.57 


.62 


.65 


.62 


.66 


.63 


.65 


.67 


.67 


8. Stimulated 
intellectual effort 


.50 


.60 


.59 




.59 


.59 


.61 


.55 


.70 


.72 


.67 


.66 


9. Encrgd multiple 
resources 


















.77 


.82 


.85 


.85 


10. Explained clearly 




.58 


.60 


.51 


















1 1 . Related to real life 


























12. Tests cover imprt. 
points 


























13. Introduce 
stimulating ideas 


.57 


.67 


.67 


.59 


.56 


.56 


.61 


.56 


.62 


.63 






14. Involved Ss in 
“hands on” 


















.63 


.64 


.69 


.73 


15. Inspired to set high 
goals 


.53 


.59 


.57 


.56 


.63 


.62 


.64 


.60 


.72 


.73 


.74 


.77 


16. Asked to share 
experiences 




.57 


.60 


.59 


.66 


.68 


.72 


.60 








.63 


17. Provided timely 
feedback 


























18. Asked Ss to help 
each other 










.58 


.60 


.62 




.63 


.63 


.65 


.71 


19. Creative 
assessments 


.52 


.61 


.63 


.50 


.72 


.76 


.78 


.77 


.66 


.68 


.65 


.74 


20. Enrgd out class S/F 
contact 




















.63 


.64 






Obj. 30. Values 
Development. 




Obj. 31. ( 
Anab 


Critical 

/sis 




Obj. 32. Interest in 
Learning 




s 


M 


L 


VL 


S 


M 


L 


VL 


s 


M 


L 


VL 


1. Displayed psnl 
interest in Ss 


.61 




.69 


.63 










.70 


.72 


.74 


.76 


2. Helped Ss answ own 
Os 


.66 


.72 


.73 


.65 


.68 


.71 


.72 


.72 


.79 


.81 


.83 


.85 


3. Scheduled work 
helpfully 


























4. Demonstrated imp of 
subject 


.62 


.70 


.75 


.67 






.65 


.63 


.71 


.72 


.75 


.74 


5. Funned icciiii5, 
discussion 


























6. Made clear how 
topics fit 


.61 


.69 


.73 


.65 








.64 


.70 


.72 


.74 




7. Explained criticisms 










.65 


.68 


.66 


.67 


.70 


.73 


.77 


.79 


8. Stimulated 
intellectual effort 


.65 


.69 






.72 


.75 


.74 


.68 


.78 


.83 


.85 


.82 
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9. Encrgd multiple 
resources 






























10. Explained clearly 






.68 












.70 








11. Related to real life 


.64 


.71 


.67 




















12. Tests cover imprt. 
points 


























13. Introduce 
stimulating ideas 


.70 


.77 


.78 


.69 


.69 


.71 


.73 


.71 


.77 


.81 


.82 


.78 


14. Involved Ss in 
“hands on” 


























15. Inspired to set high 
goals 


.66 


.71 


.69 


.61 


.68 


.69 


.67 


.64 


.78 


.80 


.81 


.81 


16. Asked to share 
experiences 


.74 


.75 


.75 


.70 


.70 


.72 


.74 


.75 






.75 




17. Provided timely 
feedback 


























18. Asked Ss to help 
each other 


.66 


.69 






.64 


.66 




.64 


.72 


.74 


.75 


.76 


19. Creative 
assessments 










.70 


.71 


.73 


.73 






.73 




20. Enrgd out class S/F 
contact 



























S=small (10-14), M=medium (15-34), L=large (35-49), VL=very large (50+) 

Only the most highly correlated items are shown. 

Note: Analyses reported in Table 7 used a more restricted data set. Classes with response rates less than 75% or not reporting 
the number enrolled were also excluded. 
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Class size is relevant in another way. Average ratings of the frequency with which each method is 
employed varies with the size of the class. These ratings also vary with the degree to which students 
were motivated (really wanted the course regardless of who taught it). Faculty members participating in 
the program want to know if their ratings were above or below average, especially on those items shown 
to be most related to progress on objectives they have chosen. 

To obtain a meaningful answer to this question, it is necessary to know the average rating for each item 
for classes grouped according to both class size and student motivation. Accordingly, four class sizes 
were identified: Small (10-14), Medium (15-34), Large (35-49), and Very Large (50 or more). 

Similarly, five “motivation” levels were established, representing roughly the upper 10 percent (High), 
the next 20 percent (High Average), the middle 40 percent (Average), the next 20 percent (Low 
Average), and the lowest 10 percent (Low). By jointly considering these two classification methods, a 4 
X 5 table was constructed consisting of 20 cells (one for each combination of class size and student 
motivation). Average scores on each of the 20 teaching methods items were then computed for each 
item. Results are shown below in Table 8. 



Table 8 

Average Scores for Method Items by Class Size and Level of Student Motivation 



1. Displayed a personal interest in students and their learning 



Student 

Motivation 

(#39) 




Class Size (' 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.29 


4.18 


4.10 


3.98 


Low Average 


4.38 


4.29 


4.17 


4.13 


Average 


4.45 


4.38 


4.29 


4.22 


High Average 


4.55 


4.45 


4.42 


4.23 


High 


4.61 


4.53 


4.44 


4.44 



2. Found ways to help students answer their own questions 



Student 

Motivation 

(#39) 




Class Size (' 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.03 


3.90 


3.83 


3.67 


Low Average 


4.12 


4.04 


3.93 


3.83 


Average 


4.20 


4.14 


4.04 


3.95 


High Average 


4.29 


4.21 


4.17 


3.97 


High 


4.36 


4.31 


4.22 


4.24 



3. Scheduled course work (class activities, tests, projects) in ways which encouraged students to stay up- 
to-date in their work 



Student 

Motivation 

(#39) 




Class Size (' 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.11 


4.07 


3.97 


3.86 


Low Average 


4.21 


4.16 


4.08 


4.02 


Average 


4.25 


4.24 


4.16 


4.09 


High Average 


4.35 


4.29 


4.24 


4 A3 


High 


4.39 


4.34 


4.23 


4.21 



4. Demonstrated the importance and significance of subject matter 



Student 




Class Size (Enrollment) 






1 1 1 
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Motivation 




Small 


Medium 


Large 


Very Large 


(#39) 


Low 


4.19 


4.09 


4.09 


4.03 


Low Average 


4.30 


4.24 


4.21 


4.18 




Average 


4.39 


4.37 


4.35 


4.30 




High Average 


4.50 


4.45 


4.47 


4.38 




High 


4.57 


4.54 


4.51 


4.53 



5. Formed teams or discussion groups to facilitate learning 



Student 

Motivation 

(#39) 




Class Size C 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.42 


3.50 


3.12 


2.85 


Low Average 


3.60 


3.58 


3.24 


2.90 


Average 


3.66 


3.68 


3.38 


3.18 


High Average 


3.75 


3.72 


3.58 


3.51 


High 


3.86 


3.84 


3.66 


3.55 



6. Made it clear how each topic fit into the course 



Student 

Motivation 

(#39) 




Class Size ( 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.04 


3.95 


3.95 


3.90 


Low Average 


4.18 


4.12 


4.10 


4.05 


Average 


4.27 


4.25 


4.23 


4.17 


High Average 


4.39 


4.34 


4.38 


4.25 


High 


4.46 


4.43 


4.40 


4.42 



7. Explained the reasons for criticisms of students’ academic performance 



Student 

Motivation 

(#39) 




Class Size ( 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.72 


3.61 


3.42 


3.31 


Low Average 


3.83 


3.73 


3.54 


3.46 


Average 


3.91 


3.84 


3.68 


3.54 


High Average 


4.02 


3.92 


3.84 


3.62 


High 


4.13 


4.08 


3.92 


3.98 



8. Stimulated students to intellectual effort beyond that required by most classes 



Student 

Motivation 

(#39) 




Class Size C 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.82 


3.64 


3.52 


3.43 


Low Average 


3.93 


3.78 


3.70 


3.63 


Average 


4.00 


3.91 


3.83 


3.75 


High Average 


4.10 


3.98 


4.00 


3.90 


High 


4.16 


4.10 


4.11 


4.17 



9. Encouraged students to use multiple resources... to improve understanding 



Student 




Class Size C 


enrollment) 


Motivation 




Small 


Medium 


Large 


Very Large 


(#39) 


Low 


3.77 


3.66 


3.39 


3.12 




Low Average 


3.88 


3.74 


3.46 


3.31 
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Average 


3.93 


3.84 


3.67 


3.40 




High Average 


4.00 


3.89 


3.84 


3.61 




High 


4.05 


3.98 


3.88 


3.97 



10. Explained course material clearly and concisely 



Student 

Motivation 

(#39) 




Class Size (1 


inrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.93 


3.89 


3.84 


3.80 


Low Average 


4.07 


4.05 


3.99 


3.97 


Average 


4.16 


4.16 


4.13 


4.10 


High Average 


4.29 


4.23 


4.25 


4.15 


High 


4.37 


4.33 


4.29 


4.30 



11. Related course material to real life situations 



Student 

Motivation 

(#39) 




Class Size (1 


Enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.03 


3.94 


4.05 


3.86 


Low Average 


4.17 


4.14 


4.16 


4.06 


Average 


4.30 


4.28 


4.31 


4.28 


High Average 


4.41 


4.35 


4.43 


4.36 


High 


4.47 


4.44 


4.45 


4.45 



12. Gave tests, pro jects, etc, that covered the most important points of the course 



Student 

Motivation 

(#39) 




Class Size (1 


Enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.14 


4.08 


4.12 


4.05 


Low Average 


4.23 


4.21 


4.25 


4.20 


Average 


4.33 


4.31 


4.33 


4.30 


High Average 


4.41 


4.36 


4.38 


4.24 


High 


4.43 


4.36 


4.32 


4.23 



13. Introduced stimulatini^ ideas about the subject 



Student 

Motivation 

(#39) 




Class Size ( 


Enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.81 


3.70 


3.72 


3.62 


Low Average 


4.00 


3.92 


3.88 


3.84 


Average 


4.13 


4.09 


4.07 


4.01 


High Average 


4.27 


4.20 


4.23 


4.10 


High 


4.36 


4.32 


4.28 


4.27 



14. Involved students in hands on projects such as research, case studies, or real life activities 



Student 

Motivation 

(#59) 




Class Size C 


Enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.73 


3.52 


3.32 


3.07 


Low Average 


3.87 


3.67 


3.36 


3.12 


Average 


4.01 


3.88 


3.64 


3.47 


High Average 


4.13 


4.03 


3.92 


3.88 


High 


4.28 


4.20 


4.02 


3.86 
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15. Inspired students to set and achieve Roals which really challenged them 



Student 

Motivation 

(#39) 




Class Size (] 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.70 


3.52 


3.28 


3.16 


Low Average 


3.83 


3.66 


3.47 


3.33 


Average 


3.92 


3.82 


3.64 


3.52 


High Average 


4.06 


3.95 


3.86 


3.75 


High 


4.21 


4.14 


4.03 


4.07 



16. Asked students to share ideas and experiences with others whose backgrounds and viewpoints differ 
from their own 



Student 

Motivation 

(#39) 




Class Size C 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.57 


3.47 


3.25 


2.94 


Low Average 


3.78 


3.64 


3.42 


3.15 


Average 


3.84 


3.79 


3.60 


3.32 


High Average 


3.96 


3.87 


3.76 


3.46 


High 


4.07 


3.98 


3.83 


3.93 



17. Provided timely and frequent feedback on tests, reports, projects, etc, to help students improve 



Student 

Motivation 

(#39) 




Class Size (' 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


4.00 


3.93 


3.89 


3.69 


Low Average 


4.13 


4.07 


3.98 


3.84 


Average 


4.18 


4.14 


4.08 


3.95 


High Average 


4.26 


4.19 


4.16 


3.89 


High 


4.32 


4.25 


4.20 


4.14 



18. Asked students to help each other understand ideas and concepts 



Student 

Motivation 

(#39) 




Class Size ( 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.71 


3.63 


3.42 


3.23 


Low Average 


3.86 


3.74 


3.53 


3.38 


Average 


3.93 


3.87 


3.66 


3.53 


High Average 


4.03 


3.95 


3.85 


3.69 


High 


4.14 


4.09 


3.93 


3.97 



19. Gave projects, tests, or assignments that required original or creative thinking 



Student 

Motivation 

(#39) 




Class Size ( 


enrollment) 


Small 


Medium 


Large 


Very Large 


Low 


3.83 


3.75 


3.47 


3.21 


Low Average 


4.00 


3.89 


3.60 


3.39 


Avera^^e 


4.07 


4.01 


3.78 


3.54 


High Average 


4.17 


4.07 


3.89 


3.67 


High 


4.24 


4.13 


3.94 


3.83 



20. Encouraged student-faculty interaction outside of class (office visits, phone calls, email, etc.) 



Student 



Class Size (Enrollment) 
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Motivation 




Small 


Medium 


Large 


Very Large 


(#39) 


Low 


3.86 


3.74 


3.64 


3.55 


Low Average 


3.96 


3.87 


3.77 


3.77 




Average 


4.03 


3.96 


3.90 


3.83 




High Average 


4.09 


3.98 


4.03 


3.78 




High 


4.14 


4.05 


4.07 


4.15 



Note: Analyses reported in Table 8 used a more restricted data set. Classes with response rates less than 75% or not reporting 
the number enrolled were also excluded. 



The information provided in these cells is intended to provide diagnostic assistance to those using the 
Diagnostic Form (see pages 4 and 5 of the sample IDEA Report included in Appendix A). This is done 
through a series of steps. 

First, “relevant” objectives are identified (those the instructor identified as “Important” or “Essential”). 
Then, the most relevant teaching methods — those most closely related to a given progress rating — are 
identified (see Table 7). The class is then classified according by its size and level of student 
motivation. Results on the “most relevant” items are then compared with those for “similar classes” 
using the data reported above. 

If the obtained mean is 0.3 (approximately one standard error) or more above the mean for similar 
classes, the user is encouraged to retain this approach; if it is 0.3 or more below the mean for similar 
classes, the user is advised to “consider increasing the frequency” with which the method is employed. 

Table 9 provides normative information for each of the items included on the Diagnostic Form. 

Separate norms for the Short Form are not included for reasons described in Section VI of this report. 
Norms are provided for all institutions and for those whose highest degree offered is the Associate (2- 
year). Baccalaureate, Master’s, or Doctoral. As noted earlier, a number of “Other” institutions also 
participated. These were principally institutions with highly specialized emphases; they were so 
heterogeneous that a meaningful norm (comparison) group could not be described. 

For items or measures that are intended to provide information about the effectiveness of instruction, 
norms are provided for both unadjusted (raw) and adjusted scores. Of these. Items 21-32 represent 
student ratings of the progress they made on each of 12 learning objectives; for these 12 items, the only 
classes included are those for which the objective was rated as “Essential” or “Important” by the 
instructor. The process of adjusting scores is described in Section III of this report. 

Table 9 also provides norms for five “scales” descriptive of alternative teaching approaches or styles 
contained in the IDEA Survey. A further description of these scales is provided in Section II of this 
report. 

As shown in Table 9, for the most part, differences among types of institutions were relatively slight. 
There appeared to be a tendency for ratings to be slightly higher at two-year institutions. For example, 

on Item 17 (frequency and timeliness of feedback) an average of 4.3 was at the 49‘*^ percentile for 2-year 
colleges but at the 61®* percentile for those offering the baccalaureate degree. Similarly, on Item 47 (use 
of educational technology), an average rating of 3.7 was equivalent to the 46**^ percentile for 2-year 
colleges but the 57**^ percentile for 4-year colleges. But there were numerous exceptions The average 
ratings for the four types of institutions, given at the bottom of each table, were very close to each other. 

Differences among types of institutions were so slight that the IDEA Center will continue to use the all- 
classes norm in its reports. Users who feel more comfortable in interpreting results if they are compared 

22 

http://www.idea.ksu.edu/users/tchrptI2/TECHNICAL%20REP0RT%20I2.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 24 of 33 



with those from similarly classified institutions will find the necessary information in the Table 9 below. 

Table 9 

Percentile Ranks for IDEA Diagnostic Form Items and Scales 
By Type of Institution 



1. Displayed personal interest 
answer own questions 



2. Helped students 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.5 


0 


0 


1 


0 


1 


2.8 


1 


1 


1 


1 


1 


2.8 


2 


1 


2 


2 


2 


3.0 


2 


1 


1 


1 


2 


3.0 


3 


2 


4 


3 


4 


3.3 


4 


3 


3 


4 


5 


3.3 


7 


5 


8 


8 


9 


3.5 


6 


5 


6 


6 


8 


3.5 


12 


9 


13 


13 


15 


3.7 


11 


9 


10 


10 


13 


3.7 


19 


15 


20 


21 


22 


3.9 


17 


15 


16 


17 


20 


3.9 


30 


25 


32 


32 


33 


4.1 


26 


23 


25 


26 


28 


4.1 


43 


37 


46 


46 


46 


4.3 


38 


35 


37 


38 


41 


4.3 


59 


53 


62 


62 


61 


4.5 


54 


52 


54 


55 


56 


4.5 


76 


73 


80 


79 


77 


4.7 


74 


73 


73 


75 


74 


4.7 


90 


89 


92 


91 


90 


4.9 


92 


92 


92 


93 


92 


4.9 


98 


97 


98 


98 


98 


5.0 


98 


98 


98 


98 


97 


5.0 


99 


99 


99 


99 


99 


Avg. 


4.3 


4.4 


4.3 


4.3 


4.3 


Avg. 


4.1 


4.2 


4.1 


4.1 


4.1 


3. Scheduled work helpfully 








4. Demonstrated significance 




Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.8 


1 


1 


1 


1 


1 


2.8 


0 


0 


0 


0 


0 


3.0 


2 


1 


2 


2 


2 


3.0 


1 


1 


1 


1 


1 


3.3 


5 


3 


5 


5 


6 


3.3 


3 


2 


3 


3 


4 


3.5 


8 


6 


9 


9 


10 


3.5 


5 


4 


6 


5 


7 


3.7 


14 


10 


15 


15 


16 


3.7 


9 


7 


10 


9 


12 


3.9 


22 


18 


24 


24 


26 


3.9 


16 


14 


17 


16 


20 


4.1 


35 


29 


37 


37 


39 


4.1 


26 


23 


27 


26 


30 


4.3 


51 


45 


53 


54 


54 


4.3 


40 


37 


42 


41 


44 


4.5 


70 


65 


73 


73 


72 


4.5 


59 


57 


60 


60 


61 


4.7 


87 


85 


89 


89 


88 


4.7 


78 


78 


80 


80 


79 


4.9 


97 


97 


98 


98 


97 


4.9 


94 


94 


95 


95 


94 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


98 


99 


99 


98 


Avg. 


4.2 


4.3 


4.2 


4.2 


4.2 


Avg. 


4.3 


4.4 


4.3 


4.3 


4.3 




5 


. Formed “teams” 










6. Made clear how topics fit 




Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


2 


2 


3 


3 


3 


1.5 


0 


0 


0 


0 


0 


2.0 


10 


9 


11 


11 


10 


2.0 


0 


0 


0 


0 


0 


2.5 


20 


20 


21 


22 


18 


2.5 


0 


0 


0 


0 


0 


2.8 


27 


28 


29 


28 


24 


2.8 


1 


1 


1 


1 


2 


3.0 


31 


33 


34 


32 


27 


3.0 


2 


2 


2 


2 


3 


3.3 


38 


40 


41 


39 


33 


3.3 


5 


5 


5 


5 


7 


3.5 


43 


47 


46 


44 


38 


3.5 


9 


9 


8 


9 


11 


3.7 


49 


53 


52 


49 


43 


3.7 


15 


14 


14 


14 


18 


3.9 


55 


59 


58 


56 


49 


3.9 


23 


23 


22 


23 


27 


4.1 


62 


66 


65 


62 


56 


4.1 


34 


34 


32 


34 


39 



er|c 
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4.3 


70 


74 


73 


70 


65 


4.3 


50 


49 


48 


50 


53 


4.5 


79 


82 


81 


79 


75 


4.5 


68 


68 


67 


69 


70 


4.7 


88 


91 


90 


87 


86 


4.7 


85 


85 


86 


87 


86 


4.9 


96 


97 


97 


96 


96 


4.9 


97 


97 


97 


97 


96 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.5 


3.5 


3.4 


3.5 


3.6 


Avg. 


4.2 


4.2 


4.2 


4.2 


4.2 




7. Explained criticisms 














8. Stimulated 


intellectual effort 




















Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


2 


2 


2 


2 


3 


2.5 


2 


1 


3 


1 


2 


2.8 


5 


5 


4 


5 


6 


2.8 


4 


3 


7 


4 


5 


3.0 


9 


8 


8 


9 


11 


3.0 


7 


6 


11 


7 


9 


3.3 


18 


17 


16 


20 


22 


3.3 


15 


12 


20 


16 


18 


3.5 


28 


27 


26 


31 


31 


3.5 


24 


20 


29 


25 


27 


3.7 


40 


38 


38 


44 


43 


3.7 


35 


30 


42 


37 


37 


3.9 


55 


52 


52 


59 


56 


3.9 


48 


44 


56 


50 


50 


4.1 


68 


66 


67 


72 


68 


4.1 


62 


57 


68 


64 


63 


4.3 


80 


79 


80 


84 


80 


4.3 


75 


73 


79 


77 


76 


4.5 


90 


89 


90 


92 


90 


4.5 


87 


86 


89 


88 


87 


4.7 


96 


96 


96 


97 


96 


4.7 


95 


94 


96 


95 


95 


4.9 


99 


99 


99 


99 


99 


4.9 


99 


98 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.8 


3.8 


3.8 


3.7 


3.8 


Avg. 


3.9 


3.9 


3.8 


3.8 


3.8 


9. Encouraged using multiple resources 












10. Explained 


clearly 
























Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


1 


0 


2 


1 


1 


2.0 


0 


0 


0 


0 


0 


2.5 


5 


3 


7 


6 


5 


2.5 


2 


1 


2 


2 


2 


2.8 


9 


6 


13 


12 


10 


2.8 


4 


2 


4 


4 


5 


3.0 


14 


10 


19 


17 


15 


3.0 


5 


3 


6 


6 


7 


3.3 


23 


18 


29 


27 


24 


3.3 


10 


7 


11 


11 


12 


3.5 


31 


26 


37 


36 


32 


3.5 


14 


10 


15 


16 


17 


3.7 


40 


36 


46 


45 


41 


3.7 


20 


15 


21 


22 


24 


3.9 


51 


47 


57 


55 


51 


3.9 


28 


22 


29 


31 


33 


4.1 


61 


58 


68 


66 


60 


4.1 


38 


31 


40 


. 41 


43 


4.3 


73 


71 


80 


76 


72 


4.3 


52 


43 


55 


55 


56 


4.5 


84 


83 


89 


86 


83 


4.5 


68 


60 


72 


71 


72 


4.7 


93 


92 


95 


94 


92 


4.7 


84 


80 


88 


87 


85 


4.9 


98 


98 


98 


98 


98 


4.9 


96 


95 


98 


97 


96 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


98 


99 


99 


99 


Avg. 


3.8 


3.9 


3.7 


3.7 


3.8 


Avg. 


4.1 


4.2 


4.1 


4.1 


4.1 


11.] 


Related to real life 










12. Tests covered important points 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


1 


1 


1 


1 


1 


2.5 


0 


0 


0 


0 


0 


2.8 


2 


2 


3 


2 


2 


2.8 


1 


1 


1 


1 


1 
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3.0 


4 


4 


4 


3 


4 


3.0 


2 


1 


2 


2 


2 


3.3 


8 


8 


9 


7 


9 


3.3 


4 


3 


5 


4 


5 


3.5 


12 


13 


14 


11 


13 


3.5 


7 


5 


8 


7 


9 


3.7 


18 


19 


20 


17 


19 


3.7 


11 


9 


13 


11 


15 


3.9 


25 


27 


29 


24 


27 


3.9 


19 


15 


21 


19 


24 


4.1 


34 


36 


39 


33 


36 


4.1 


28 


23 


31 


29 


35 


4.3 


46 


49 


51 


45 


48 


4.3 


42 


36 


46 


43 


49 


4.5 


61 


63 


64 


60 


62 


4.5 


60 


53 


65 


62 


67 


4.7 


77 


79 


78 


77 


77 


4.7 


80 


75 


84 


82 


84 


4.9 


93 


94 


93 


93 


93 


4.9 


95 


94 


97 


96 


96 


5.0 


98 


98 


98 


98 


98 


5.0 


99 


98 


99 


99 


99 


Avg. 


4.2 


4.2 


4.2 


4.2 


4.2 


Avg. 


4.3 


4.4 


4.2 


4.3 


4.2 



13. Introduced stimulating ideas 14. Involved in 



“hands on” 



Mean 


All : 


2-Yr BA,BS MA,MS Doct. 


Mean 


All : 


2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


3 


2 


2 


3 


3 


2.5 


1 


1 


1 


1 


2 


2.5 


8 


8 


7 


10 


9 


2.8 


3 


3 


4 


3 


4 


2.8 


13 


13 


12 


14 


14 


3.0 


5 


5 


6 


5 


7 


3.0 


18 


18 


16 


19 


18 


3.3 


11 


10 


12 


11 


14 


3.3 


25 


27 


24 


27 


25 


3.5 


17 


15 


18 


17 


20 


3.5 


32 


34 


32 


33 


32 


3.7 


25 


22 


26 


26 


28 


3.7 


40 


42 


40 


41 


39 


3.9 


35 


33 


37 


37 


39 


3.9 


49 


52 


50 


50 


47 


4.1 


48 


45 


50 


50 


50 


4.1 


59 


62 


60 


60 


56 


4.3 


62 


60 


64 


64 


63 


4.3 


69 


73 


71 


70 


66 


4.5 


77 


76 


79 


79 


76 


4.5 


80 


84 


81 


81 


78 


4.7 


89 


89 


91 


90 


88 


4.7 


90 


92 


91 


90 


88 


4.9 


97 


97 


98 


98 


97 


4.9 


97 


98 


97 


97 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


4.0 


4.1 


4.0 


4.0 


4.0 


Avg. 


3.7 


3.7 


3.8 


3.7 


3.8 



15. Inspired ambitious goals 
students to share ideas 

Mean All 2-Yr BA,BS MA,MS Doct. 


Mean 


All : 


16. Asked diverse 
2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


2 


2 


3 


3 


3 


2.5 


3 


2 


3 


3 


3 


2.5 


9 


7 


10 


11 


10 


2.8 


7 


6 


7 


7 


8 


2.8 


15 


13 


16 


18 


15 


3.0 


12 


10 


11 


12 


14 


3.0 


20 


18 


22 


24 


20 


3.3 


22 


18 


23 


24 


24 


3.3 


29 


27 


31 


33 


28 


3.5 


31 


27 


33 


35 


34 


3.5 


36 


35 


39 


40 


34 


3.7 


42 


38 


45 


47 


44 


3.7 


44 


43 


48 


49 


42 


3.9 


55 


51 


57 


60 


56 


3.9 


54 


54 


58 


57 


50 


4.1 


67 


63 


69 


72 


67 


4.1 


63 


64 


68 


66 


60 


4.3 


79 


76 


80 


82 


79 


4.3 


74 


75 


78 


' 76 


70 


4.5 


88 


87 


90 


90 


88 


4.5 


84 


85 


87 


85 


80 


4.7 


95 


94 


96 


96 


95 


4.7 


92 


93 


94 


93 


90 


4.9 


99 


98 


99 


99 


98 


4.9 


98 


98 


99 


98 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.7 


3.8 


3.7 


3.7 


3.7 


Avg. 


3.7 


3.7 


3.6 


3.6 


3.7 



17. Timely feedback 




25 
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students to help others 



Mean 


All : 


2-Yr 


BA,BS 


MA.MS Doct. 


Mean 


All : 


2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


1 


0 


2 


2 


2 


2.5 


3 


3 


2 


3 


4 


2.8 


3 


2 


4 


4 


4 


2.8 


8 


7 


7 


8 


8 


3.0 


5 


3 


7 


6 


7 


3.0 


12 


11 


11 


13 


13 


3.3 


10 


6 


12 


11 


12 


3.3 


21 


20 


21 


23 


22 


3.5 


14 


10 


17 


16 


18 


3.5 


30 


28 


31 


32 


30 


3.7 


21 


16 


24 


22 


25 


3.7 


40 


38 


42 


43 


40 


3.9 


29 


24 


35 


31 


35 


3.9 


52 


50 


55 


55 


51 


4.1 


40 


34 


46 


41 


46 


4.1 


64 


62 


68 


67 


63 


4.3 


54 


49 


61 


56 


60 


4.3 


76 


74 


80 


78 


75 


4.5 


71 


67 


77 


73 


74 


4.5 


87 


86 


91 


88 


86 


4.7 


86 


84 


90 


87 


87 


4.7 


94 


94 


97 


95 


94 


4.9 


97 


96 


98 


97 


97 


4.9 


98 


98 


99 


99 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


4.1 


4.2 


4.0 


4.1 


4.0 


Avg. 


3.8 


3.8 


3.8 


3.7 


3.8 



19. Required originality 
out-of-class contact 

Mean All 2-Yr BA,BS MA,MS Doct. 


Mean 


All : 


20. Encouraged 
2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


3 


2 


3 


3 


3 


2.5 


2 


3 


4 


2 


2 


2.8 


6 


5 


7 


7 


7 


2.8 


5 


7 


7 


4 


. 5 


3.0 


10 


8 


11 


11 


10 


3.0 


9 


11 


12 


7 


9 


3.3 


17 


15 


18 


20 


18 


3.3 


16 


19 


21 


14 


16 


3.5 


24 


22 


25 


27 


25 


3.5 


24 


27 


30 


21 


24 


3.7 


32 


30 


33 


35 


33 


3.7 


33 


37 


39 


30 


33 


3.9 


43 


41 


44 


46 


42 


3.9 


44 


49 


51 


42 


44 


4.1 


54 


53 


55 


58 


52 


4.1 


56 


61 


62 


54 


56 


4.3 


67 


66 


67 


70 


64 


4.3 


69 


73 


74 


68 


68 


4.5 


80 


80 


80 


82 


77 


4.5 


82 


85 


85 


81 


81 


4.7 


90 


91 


90 


91 


89 


4.7 


92 


93 


93 


92 


91 


4.9 


97 


98 


97 


98 


97 


4.9 


98 


98 


98 


98 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.9 


4.0 


3.9 


3.9 


3.9 


Avg. 


3.9 


3.8 


3.8 


3.9 


3.9 



21. Factual knowledge (unadjusted) 
knowledge (adjusted) 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


1 


0 


0 


2.8 


2 


1 


3 


1 


2 


3.0 


3 


3 


5 


3 


4 


3.3 


8 


7 


12 


8 


9 


3.5 


15 


12 


19 


15 


16 


3.7 


24 


20 


29 


25 


25 


3.9 


37 


34 


43 


39 


39 


4.1 


53 


49 


57 


55 


54 


4.3 


70 


68 


73 


71 


70 


4.5 


85 


84 


86 


86 


85 


4.7 


94 


94 


95 


95 


95 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 







21. Factual 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


1 


1 


2 


1 


1 


2.8 


3 


2 


5 


3 


3 


3.0 


5 


4 


8 


5 


6 


3.3 


11 


9 


16 


11 


12 


3.5 


18 


16 


24 


18 


20 


5.7 


28 


26 


^ y' 
JU 


/> o 


y\rs 


3.9 


42 


40 


48 


42 


43 


4.1 


58 


57 


63 


58 


59 


4.3 


74 


74 


77 


73 


74 


4.5 


87 


87 


90 


87 


87 


4.7 


95 


95 


96 


95 


95 


4.9 


98 


98 


98 


98 


98 


5.0 


99 


99 


99 


99 


99 
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Avg. 4.0 4.0 3.9 4.0 4.0 

22. Principles, theories (unadjusted) 
theories (adjusted) 

Mean All 2-Yr BA.BS MA.MS Doct. 


Avg. 

Mean 


4.0 

All 
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4.0 3.9 4.0 4.0 

22. Principles, 

2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


1 


0 


0 


2.5 


1 


1 


2 


1 


1 


2.8 


2 


1 


3 


2 


2 


2.8 


3 


2 


6 


3 


3 


3.0 


4 


3 


6 


4 


4 


3.0 


5 


4 


9 


5 


6 


3.3 


10 


7 


14 


10 


10 


3.3 


12 


10 


19 


12 


13 


3.5 


17 


13 


22 


17 


18 


3.5 


20 


17 


27 


20 


22 


3.7 


27 


23 


33 


27 


28 


3.7 


32 


28 


39 


31 


33 


3.9 


42 


38 


47 


42 


43 


3.9 


47 


44 


53 


46 


47 


4.1 


58 


55 


63 


59 


58 


4.1 


63 


61 


69 


62 


63 


4.3 


75 


73 


78 


76 


74 


4.3 


79 


78 


82 


79 


78 


4.5 


89 


88 


89 


90 


88 


4.5 


90 


90 


91 


90 


89 


4.7 


96 


96 


96 


97 


96 


4.7 


96 


96 


97 


96 


96 


4.9 


99 


99 


99 


99 


99 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.9 


4.0 


3.9 


3.9 


3.9 


Avg. 


3.9 


4.0 


3.8 


3.9 


3.9 


23. Applications (unadjusted) 












23. Applications 


(adjusted) 
























Mean 


All 


2-Yr BA,BS MA,MS Doct. 


Mean 


All 


2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


1 


0 


0 


2.5 


1 


1 


2 


1 


1 


2.8 


2 


1 


3 


2 


2 


2.8 


3 


2 


5 


3 


4 


3.0 


4 


3 


5 


4 


5 


3.0 


6 


4 


8 


6 


6 


3.3 


10 


7 


11 


10 


11 


3.3 


12 


11 


17 


13 


14 


3.5 


16 


13 


20 


17 


18 


3.5 


20 


18 


26 


20 


22 


3.7 


26 


23 


30 


27 


28 


3.7 


31 


29 


36 


30 


32 


3.9 


39 


36 


44 


40 


40 


3.9 


44 


44 


49 


44 


45 


4.1 


54 


52 


57 


55 


54 


4.1 


59 


61 


64 


59 


59 


4.3 


69 


69 


71 


71 


69 


4.3 


74 


76 


77 


73 


73 


4.5 


84 


84 


85 


85 


83 


4.5 


86 


88 


88 


85 


85 


4.7 


93 


94 


94 


94 


93 


4.7 


94 


95 


95 


94 


93 


4.9 


98 


98 


98 


99 


98 


4.9 


98 


98 


98 


98 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


98 


Avg. 


4.0 


4.0 


3.9 


4.0 


4.0 


Avg. 


4.0 


4.0 


3.9 


4.0 


4.0 


24. Professional skills, attitudes (unadjusted) 










24. Professional 


skills, attitudes (adjusted) 




















Mean 


All 


2-Yr BA,BS MA,MS Doct. 


Mean 


All 


2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


1 


0 


0 


2.5 


1 


1 


2 


1 


1 


2.8 


2 


2 


2 


2 


2 


2.8 


3 


3 


4 


3 


3 


3.0 


4 


3 


4 


*4 


4 


3.0 


5 


5 


7 


5 


6 


3.3 


9 


8 


10 


9 


11 


3.3 


11 


11 


14 


11 


13 


3.5 


15 


14 


16 


15 


18 


3.5 


18 


19 


21 


18 


20 


3.7 


23 


22 


25 


24 


27 


3.7 


28 


29 


31 


27 


30 


3.9 


35 


33 


37 


36 


39 


3.9 


41 


44 


43 


39 


43 


4.1 


48 


47 


49 


50 


52 


4.1 


56 


60 


57 


53 


57 
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4.3 


64 


63 


65 


66 


67 


4.3 


71 


75 


71 


68 


71 


4.5 


80 


80 


79 


81 


81 


4.5 


84 


87 


84 


82 


83 


4.7 


91 


91 


91 


92 


92 


4.7 


92 


94 


93 


92 


92 


4.9 


98 


98 


97 


98 


98 


4.9 


97 


98 


97 


97 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


99 


98 


98 


98 


Avg. 


4.0 


4.1 


4.0 


4.0 


4.0 


Avg. 


4.0 


4.0 


4.0 


4.0 


4.0 



25. Team skills (unadjusted) 
(adjusted) 
















25. Team skills 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


1 


1 


1 


1 


1 


2.5 


3 


3 


4 


3 


2 


2.5 


4 


5 


7 


3 


4 


2.8 


6 


7 


8 


5 


5 


2.8 


7 


9 


11 


6 


7 


3.0 


8 


10 


11 


7 


8 


3.0 


11 


12 


16 


9 


11 


3.3 


15 


17 


18 


14 


15 


3.3 


19 


20 


24 


17 


19 


3.5 


21 


24 


23 


21 


22 


3.5 


26 


28 


30 


24 


28 


3.7 


30 


32 


31 


29 


32 


3.7 


35 


38 


38 


33 


38 


3.9 


41 


44 


41 


40 


44 


3.9 


47 


51 


51 


45 


49 


4.1 


54 


56 


55 


53 


57 


4.1 


61 


64 


65 


58 


63 


4.3 


68 


70 


69 


68 


70 


4.3 


75 


78 


77 


72 


77 


4.5 


81 


83 


81 


81 


83 


4.5 


86 


89 


87 


85 


87 


4.7 


92 


93 


92 


92 


92 


4.7 


93 


95 


94 


92 


94 


4.9 


98 


98 


98 


98 


98 


4.9 


97 


98 


98 


97 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


98 


99 


98 


99 


Avg. 


3.9 


3.9 


3.9 


3.9 


3.9 


Avg. 


3.9 


3.8 


3.8 


3.9 


3.9 



26. Creative capacities (unadjusted) 
capacities (adjusted) 












26. Creative 




Mean 


All 


2-Yr 


BA,BS MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


1 


0 


1 


2 


2 


2.0 


1 


1 


2 


2 


1 


2.5 


4 


3 


4 


6 


6 


2.5 


5 


3 


6 


7 


7 


2.8 


8 


5 


8 


10 


11 


2.8 


9 


7 


10 


12 


11 


3.0 


12 


8 


12 


15 


15 


3.0 


13 


11 


15 


15 


15 


3.3 


19 


16 


19 


23 


22 


3.3 


21 


18 


24 


24 


23 


3.5 


26 


23 


26 


29 


28 


3.5 


29 


27 


30 


32 


30 


3.7 


34 


33 


34 


37 


36 


3.7 


38 


37 


39 


41 


39 


3.9 


45 


45 


45 


46 


46 


3.9 


48 


49 


49 


50 


50 


4.1 


56 


57 


56 


56 


57 


4.1 


60 


62 


60 


60 


61 


4.3 


68 


70 


69 


68 


69 


4.3 


72 


74 


72 


71 


73 


4.5 


81 


82 


81 


80 


82 


4.5 


83 


85 


82 


81 


83 


4.7 


91 


93 


91 


89 


92 


4.7 


91 


93 


90 


89 


91 


4.9 


97 


98 


98 


97 


98 


4.9 


96 


97 


96 


95 


95 


5.0 


99 


99 


99 


99 


99 


5.0 


97 


98 


97 


97 


96 


Avg. 


3.9 


3.9 


3.9 


3.8 


3.8 


Avg. 


3.9 


3.9 


3.8 


3.8 


3.8 



1 

education (adjusted) 

Mean All 2-Yr BA.BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


07 RroflH libCTEl 
MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


1 


0 


0 


2.0 


1 


0 


2 


2 


2 


2.0 


2 


1 


5 


3 


3 


2.5 


7 


3 


8 


8 


8 


2.5 


8 


4 


12 


8 


10 


2.8 


13 


7 


15 


15 


15 


2.8 


14 


8 


18 


15 


17 
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3.0 


18 


11 


20 


20 


20 


3.0 


20 


14 


25 


20 


23 


3.3 


28 


20 


30 


30 


30 


3.3 


30 


25 


37 


31 


32 


3.5 


36 


29 


40 


38 


38 


3.5 


39 


34 


44 


39 


41 


3.7 


45 


39 


48 


46 


47 


3.7 


49 


45 


52 


48 


49 


3.9 


56 


51 


58 


56 


56 


3.9 


59 


57 


61 


58 


59 


4.1 


65 


62 


67 


66 


65 


4.1 


69 


68 


71 


69 


69 


4.3 


76 


75 


77 


77 


75 


4.3 


79 


80 


79 


78 


77 


4.5 


86 


87 


87 


86 


85 


4.5 


87 


88 


87 


86 


86 


4.7 


94 


95 


95 


94 


93 


4.7 


93 


94 


93 


92 


93 


4.9 


98 


99 


99 


98 


98 


4.9 


97 


98 


97 


96 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


98 


98 


97 


98 


Avg. 


3.7 


3.8 


3.6 


3.7 


3.7 


Avg. 


3.7 


3.8 


3.6 


3.7 


3.7 



28. Communication skills (unadjusted) 
Communication skills (adjusted) 

Mean All 2-Yr BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


28. 

MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


1 


0 


0 


2.0 


1 


0 


2 


1 


1 


2.0 


1 


1 


3 


2 


2 


2.5 


4 


3 


6 


5 


5 


2.5 


5 


4 


9 


5 


6 


2.8 


9 


7 


11 


10 


9 


2.8 


10 


8 


14 


11 


10 


3.0 


13 


11 


16 


14 


13 


3.0 


14 


11 


21 


16 


14 


3.3 


21 


17 


27 


23 


20 


3.3 


24 


20 


33 


27 


23 


3.5 


29 


25 


37 


32 


28 


3.5 


33 


28 


42 


36 


32 


3.7 


39 


35 


46 


42 


37 


3.7 


43 


38 


51 


47 


41 


3.9 


50 


47 


56 


54 


47 


3.9 


54 


50 


61 


57 


52 


4.1 


62 


59 


66 


64 


59 


4.1 


66 


63 


70 


68 


63 


4.3 


75 


73 


76 


77 


71 


4.3 


77 


75 


78 


79 


74 


4.5 


86 


86 


86 


87 


84 


4.5 


86 


86 


87 


87 


85 


4.7 


94 


95 


94 


95 


93 


4.7 


93 


94 


93 


94 


92 


4.9 


99 


99 


98 


99 


98 


4.9 


97 


97 


97 


97 


97 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


98 


98 


98 


98 


Avg. 


3.8 


3.8 


3.7 


3.7 


3.8 


Avg. 


3.8 


3.9 


3.7 


3.7 


3.8 



29. Find, i 


lise resources (unadjusted) 














29. Find, use 


resources 

Mean 


(adjusted) 

All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


1 


0 


0 


2.0 


0 


0 


3 


0 


0 


2.5 


2 


1 


6 


2 


2 


2.5 


3 


1 


10 


3 


3 


2.8 


6 


3 


13 


7 


6 


2.8 


8 


4 


19 


9 


7 


3.0 


10 


7 


19 


12 


11 


3.0 


12 


8 


25 


14 


13 


3.3 


22 


16 


33 


24 


24 


3.3 


24 


17 


40 


27 


25 


3.5 


32 


25 


45 


35 


34 


3.5 


35 


27 


52 


39 


37 


3.7 


44 


38 


57 


48 


45 


3.7 


47 


40 


64 


52 


48 


3.9 


58 


54 


70 


62 


59 


3.9 


61 


56 


75 


65 


61 


4.1 


71 


68 


81 


74 


71 


4.1 


74 


71 


85 


77 


72 


4.3 


84 


82 


90 


85 


83 


4.3 


85 


85 


91 


86 


84 


4.5 


92 


92 


95 


93 


91 


4.5 


92 


92 


95 


93 


92 


4.7 


97 


97 


98 


97 


97 


4.7 


97 


97 


97 


97 


96 


4.9 


99 


99 


99 


99 


99 


4.9 


99 


99 


99 


98 


99 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.7 


3.8 


3.5 


3.7 


3.7 


Avg. 


3.7 


3.8 


3.4 


3.7 


3.7 



30. Values development (unadjusted) 
development (adjusted) 

Mean All 2-Yr BA,BS MA,MS Doct. 


Mean 


30. Values 

All 2-Yr BA.BS MA,MS Doct. 


1.5 0 0 0 0 0 


1.5 


0 0 0 0 0 



ERIC 29 

'http://www.idea.ksu.edu/users/tchrptl2/TECHNICAL%20REP0RT%2012.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 31 of 34 



2.0 


0 


0 


1 


0 


0 


2.0 


1 


0 


2 


1 


1 


2.5 


3 


2 


5 


3 


4 


2.5 


4 


3 


9 


4 


6 


2.8 


7 


5 


10 


7 


9 


2.8 


9 


6 


15 


9 


11 


3.0 


11 


8 


15 


11 


13 


3.0 


14 


10 


21 


13 


16 


3.3 


21 


16 


26 


21 


22 


3.3 


24 


19 


32 


24 


26 


3.5 


30 


25 


35 


30 


31 


3.5 


33 


29 


41 


32 


35 


3.7 


40 


37 


45 


40 


41 


3.7 


45 


42 


51 


43 


46 


3.9 


53 


51 


56 


51 


53 


3.9 


57 


58 


61 


55 


57 


4.1 


65 


65 


67 


63 


63 


4.1 


70 


72 


69 


67 


69 


4.3 


77 


80 


79 


75 


75 


4.3 


81 


84 


80 


78 


80 


4.5 


88 


90 


89 


87 


87 


4.5 


89 


93 


88 


86 


89 


4.7 


95 


96 


96 


94 


94 


4.7 


95 


97 


94 


93 


95 


4.9 


99 


99 


99 


99 


99 


4.9 


98 


98 


97 


97 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


98 


98 


99 



Avg. 3.8 3.8 3.7 3.8 

31. Critical analysis (unadjusted) 
analysis (adjusted) 

Mean All 2-Yr BA.BS MA.MS 


3.8 

Doct. 


Avg. 

Mean 


3.8 

All 


3.8 

2-Yr 


3.7 

BA,BS 


3.8 

31. Critical 
MA,MS 


3.7 

Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


1 


0 


0 


2.0 


0 


0 


2 


0 


0 


2.5 


2 


1 


4 


2 


2 


2.5 


3 


1 


6 


3 


3 


2.8 


5 


3 


9 


6 


6 


2.8 


7 


4 


12 


7 


8 


3.0 


9 


6 


12 


10 


10 


3.0 


10 


7 


16 


11 


12 


3.3 


17 


13 


20 


18 


19 


3.3 


20 


15 


26 


21 


23 


3.5 


25 


20 


29 


27 


28 


3.5 


28 


23 


36 


29 


31 


3.7 


35 


30 


40 


37 


37 


3.7 


40 


35 


47 


41 


41 


3.9 


48 


45 


53 


49 


49 


3.9 


53 


50 


60 


53 


53 


4.1 


62 


59 


65 


63 


61 


4.1 


67 


66 


72 


66 


66 


4.3 


76 


75 


78 


76 


74 


4.3 


80 


80 


81 


79 


80 


4.5 


88 


88 


89 


87 


87 


4.5 


90 


91 


90 


88 


89 


4.7 


95 


95 


96 


95 


95 


4.7 


95 


96 


95 


94 


96 


4.9 


99 


99 


99 


99 


99 


4.9 


98 


98 


98 


98 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 3.8 3.9 3.8 

32. Interest in learning (unadjusted) 
learning (adjusted) 


3.8 


3.8 


Avg. 


3.8 


3.9 


3.7 


3.8 3.8 

32. Interest in 


Mean 


All 


2-Yr 


BA,BS MA.MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


1 


0 


0 


2.5 


2 


0 


3 


2 


2 


2.5 


3 


1 


6 


3 


2 


2.8 


5 


3 


7 


5 


6 


2.8 


6 


4 


12 


7 


7 


3.0 


9 


5 


12 


10 


11 


3.0 


11 


6 


17 


11 


12 


3.3 


18 


11 


23 


20 


22 


3.3 


21 


14 


31 


23 


24 


3.5 


28 


19 


33 


31 


32 


3.5 


31 


23 


42 


33 


33 


3.7 


40 


30 


47 


42 


44 


3.7 


43 


35 


54 


45 


45 


3.9 


54 


44 


61 


56 


57 


3.9 


57 


50 


66 


58 


58 


4.1 


67 


60 


74 


70 


69 


4.1 


71 


66 


78 


72 


70 


4.3 


80 


75 


85 


82 


80 


4.3 


82 


80 


88 


83 


82 


4.5 


90 


88 


94 


91 


90 


4.5 


91 


90 


93 


90 


90 




• ^ 








r\ ^ 




f\r 


r\r 


rv^ 


n/: 


AC 


4./ 


VO 


VO 


VO 


y 1 


y\i 


/ 




y\j 


y / 


u\j 




4.9 


99 


99 


99 


99 


99 


4.9 


98 


98 


99 


98 


98 


5.0 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.8 


3.9 


3.7 


3.8 


3.8 


Avg. 


3.8 


3.9 


3.6 


3.8 


3.8 
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Progress on Relevant Objectives (unadjusted) Progress on Relevant 

Objectives (adjusted) 

(PRO ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10.) 



Mean 


All 


2-Yr BA.BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


25 


0 


0 


1 




0 


0 


25 


0 


0 


1 


0 


0 


30 


2 


2 


3 




2 


2 


30 


2 


1 


3 


1 


2 


35 


5 


4 


6 




5 


5 


35 


4 


4 


6 


4 


5 


40 


11 


10 


13 




12 


12 


40 


10 


9 


14 


11 


11 


43 


18 


15 


21 




18 


20 


43 


17 


15 


22 


17 


18 


45 


24 


21 


27 




24 


25 


45 


22 


21 


28 


23 


23 


48 


34 


32 


38 




36 


35 


48 


34 


33 


41 


34 


34 


50 


43 


41 


47 




44 


43 


50 


43 


42 


49 


43 


43 


53 


57 


55 


61 




59 


57 


53 


58 


58 


63 


58 


57 


55 


67 


66 


71 




69 


67 


55 


68 


68 


72 


68 


67 


58 


81 


81 


83 




82 


79 


58 


81 


82 


84 


81 


80 


60 


88 


88 


89 




89 


87 


60 


88 


89 


90 


88 


87 


62 


93 


93 


94 




94 


93 


62 


93 


94 


94 


93 


92 


65 


98 


98 


98 




98 


98 


65 


97 


97 


97 


97 


97 


70 


99 


99 


99 




99 


99 


70 


99 


99 


99 


99 


99 


Avg. 


50.7 


51.3 


50.0 




50.5 


50.8 


Avg. 


50.9 


51.2 


49.7 


51.1 


51.0 


33. Amount of reading 




















34. Amount of 


other work 


























Mean All 


1 2-Yr 


BA,BS 


MA,MS 


Doct. 




Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


2 1 


5 




1 


1 




1.5 


0 


0 


0 


0 


0 


2.0 


6 5 


14 




6 


5 




2.0 


0 


0 


2 


1 


1 


2.5 16 16 


28 




14 


15 




2.5 


5 


3 


8 


7 


5 


2.8 26 26 


38 




23 


25 




2.8 


13 


8 


16 


17 


13 


3.0 35 34 


47 




32 


35 




3.0 


23 


16 


26 


27 


22 


3.3 53 51 


60 




51 


53 




3.3 


41 


34 


44 


46 


42 


3.5 65 64 


68 




64 


66 




3.5 


55 


50 


56 


59 


56 


3.7 75 74 


76 




74 


75 




3.7 


68 


65 


67 


71 


68 


3.9 83 82 


82 




83 


83 




3.9 


79 


78 


78 


80 


79 


4.1 88 88 


88 




89 


89 




4.1 


86 


87 


86 


87 


87 


4.3 93 93 


92 




93 


93 




4.3 


92 


93 


91 


92 


92 


4.5 96 96 


95 




96 


97 




4.5 


96 


96 


95 


96 


96 


4.7 98 98 


98 




98 


98 




4.7 


98 


99 


97 


98 


98 


4.9 99 99 


99 




99 


99 




4.9 


99 


99 


99 


99 


99 


5.0 99 99 


99 




99 


99 




5.0 


99 


99 


99 


99 


99 


Avg. 3.2 3.2 


3.0 




3.2 


3.2 




Avg. 


3.4 


3.5 


3.4 


3.4 


3.4 



35. Difficulty 

desire to take course 
Mean All 2-Yr 


BA,BS 


MA,MS 


Doct. 




Mean 


All 


2-Yr 


BA,BS 


36. Strong 
MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 




1.5 


0 


0 


0 


0 


0 


2.0 


1 


0 


2 


0 


1 




2.0 


0 


0 


0 


0 


1 


2.5 


5 


3 


7 


5 


6 




2.5 


4 


3 


5 


5 


5 


2.8 


12 


9 


16 


13 


14 




2.8 


11 


7 


11 


11 


12 


3.0 


22 


19 


27 


23 


25 




3.0 


17 


13 


17 


17 


18 


3.3 


43 


40 


47 


43 


46 




3.3 


29 


25 


28 


30 


31 


3.5 


57 


56 


60 


56 


61 




3.5 


39 


35 


37 


41 


40 


3.7 


69 


69 


72 


08 


73 




3.7 


50 


45 


43 


52 


e ^ 


3.9 


79 


80 


81 


77 


83 




3.9 


61 


56 


59 


64 


62 


4.1 


86 


87 


88 


85 


89 




4.1 


71 


66 


69 


74 


72 


4.3 


92 


93 


92 


91 


94 




4.3 


80 


75 


79 


84 


82 


4.5 


96 


96 


96 


95 


97 




4.5 


89 


84 


88 


91 


90 


4.7 


98 


98 


98 


98 


99 




4.7 


94 


91 


94 


96 


96 


4.9 


99 


99 


99 


99 


99 




4.9 


98 


97 


98 


99 


99 
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5.0 


99 


99 


99 


99 


99 


Avg. 3.4 3.5 

37. Worked hard 


3.3 


3.4 


3.4 


instructor 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


3 


1 


5 


3 


3 


2.8 


8 


4 


13 


9 


9 


3.0 


15 


9 


21 


16 


16 


3.3 


30 


23 


38 


33 


33 


3.5 


44 


37 


52 


47 


47 


3.7 


58 


53 


64 


61 


61 


3.9 


72 


68 


75 


74 


74 


4.1 


82 


79 


83 


83 


85 


4.3 


89 


88 


90 


90 


92 


4.5 


95 


93 


95 


95 


96 


4.7 


98 


97 


97 


98 


98 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 3.6 3.7 

39. Wanted course 


3.5 


3.5 


3.5 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


1 


1 


2.5 


7 


4 


7 


8 


7 


2.8 


17 


12 


18 


20 


17 


3.0 


27 


21 


28 


31 


28 


3.3 


47 


38 


48 


52 


48 


3.5 


62 


52 


62 


67 


63 


3.7 


74 


65 


75 


80 


75 


3.9 


84 


77 


85 


89 


86 


4.1 


91 


85 


92 


94 


93 


4.3 


95 


91 


96 


97 


97 


4.5 


98 


96 


98 


99 


99 


4.7 


99 


98 


99 


99 


99 


4.9 


99 


99 


99 


. 99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.3 


3.5 


3.3 


3.2 


3.3 
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5.0 


99 


99 


99 


99 


99 


Avg. 


3.7 


3.8 


3.7 


3.6 


3.6 



Mean 


38. Wanted 

All 2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


1 


1 


1 


1 


1 


2.5 


8 


9 


8 


8 


9 


2.8 


19 


20 


18 


18 


21 


3.0 


28 


31 


27 


27 


31 


3.3 


45 


47 


42 


43 


47 


3.5 


56 


59 


54 


54 


58 


3.7 


66 


69 


64 


65 


69 


3.9 


75 


77 


74 


74 


77 


4.1 


83 


84 


81 


82 


84 


4.3 


89 


90 


87 


89 


90 


4.5 


94 


94 


93 


93 


94 


4.7 


97 


97 


96 


97 


97 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.4 


3.4 


3.4 


3.4 


3.4 



40. Increased positive attitude (unadjusted) 



40. Increased 



positive attitude (adjusted) 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


2 


1 


2 


2 


2 


2.8 


5 


4 


0 


5 


6 


3.0 


9 


7 


9 


9 


10 


3.3 


17 


15 


17 


18 


19 


3.5 


25 


23 


25 


26 


27 


3.7 


35 


32 


35 


37 


36 


3.9 


47 


45 


46 


50 


48 


4.1 


60 


58 


60 


63 


61 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


1 


1 


2.5 


3 


3 


3 


3 


4 


2.8 


7 


7 


7 


7 


S 


3.0 


11 


11 


11 


11 


13 


3.3 


21 


21 


20 


20 


23 


3.5 


30 


31 


30 


29 


32 


3.7 


41 


43 


41 


39 


43 


3.9 


54 


57 


53 


52 


55 


4.1 


67 


69 


66 


65 


67 
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4.3 


74 


72 


74 


76 


74 


4.3 


78 


81 


78 


77 


78 


4.5 


85 


84 


86 


88 


85 


4.5 


87 


89 


87 


86 


87 


4.7 


94 


93 


94 


95 


94 


4.7 


93 


94 


94 


93 


93 


4.9 


98 


98 


99 


99 


99 


4.9 


97 


97 


97 


97 


96 


5.0 


99 


99 


99 


99 


99 


5.0 


98 


98 


98 


98 


97 


Avg. 


3.9 


3.9 


3.9 


3.8 


3.8 


Avg. 


3.8 


3.8 


3.8 


3.9 


3.8 



41. Excellent teacher (unadjusted) 

teacher (adjusted) 

Mean All 2-Yr BA.BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


41. Excellent 
MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 




0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


1 


0 




0 


2.5 


2 


1 


2 


2 


2 


2.5 


2 


1 


3 


2 




3 


2.8 


4 


3 


5 


4 


5 


2.8 


4 


3 


6 


4 




6 


3.0 


6 


4 


7 


6 


7 


3.0 


6 


5 


8 


7 




8 


3.3 


10 


7 


11 


11 


12 


3.3 


11 


9 


13 


12 




14 


3.5 


14 


10 


15 


15 


17 


3.5 


16 


13 


17 


17 




19 


3.7 


19 


15 


20 


21 


23 


3.7 


22 


18 


24 


22 




26 


3.9 


27 


22 


28 


28 


30 


3.9 


29 


25 


31 


30 




34 


4.1 


35 


30 


36 


37 


40 


4.1 


40 


35 


41 


40 




44 


4.3 


47 


41 


47 


49 


52 


4.3 


52 


48 


53 


52 




57 


4.5 


61 


56 


62 


63 


64 


4.5 


67 


64 


67 


66 




70 


4.7 


77 


73 


78 


79 


79 


4.7 


81 


80 


82 


80 




84 


4.9 


93 


92 


94 


93 


94 


4.9 


92 


91 


93 


91 




94 


5.0 


98 


97 


98 


98 


98 


5.0 


96 


95 


96 


95 




96 


Avg. 


4.2 


4.3 


4.2 


4.2 


4.1 


Avg. 


4.2 


4.2 


4.1 


4.2 




4.1 


42. Excellent course (unadjusted) 
















42. Excellent 




course (adjusted) 
























Mean 


All 


2-Yr BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


1.5 


0 


0 


0 


0 




0 


2.0 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 




1 


2.5 


2 


1 


2 


2 


3 


2.5 


3 


2 


3 


3 




4 


2.8 


5 


3 


5 


5 


6 


2.8 


6 


4 


7 


6 




8 


3.0 


8 


5 


8 


8 


10 


3.0 


10 


8 


10 


10 




13 


3.3 


15 


11 


15 


17 


19 


3.3 


18 


15 


19 


19 




23 


3.5 


23 


17 


23 


24 


27 


3.5 


26 


23 


27 


27 




31 


3.7 


32 


25 


32 


34 


36 


3.7 


36 


34 


37 


36 




41 


3.9 


43 


37 


43 


46 


47 


3.9 


48 


47 


49 


48 




53 


4.1 


56 


50 


56 


59 


59 


4.1 


61 


60 


61 


60 




65 


4.3 


69 


65 


69 


72 


72 


4.3 


74 


73 


73 


73 




76 


4.5 


82 


80 


82 


84 


83 


4.5 


84 


84 


84 


83 




86 


4.7 


92 


91 


92 


93 


92 


4.7 


92 


91 


92 


91 




92 


4.9 


98 


98 


98 


98 


98 


4.9 


96 


96 


97 


96 




96 


5.0 


99 


99 


99 


99 


99 


5.0 


97 


97 


98 


97 




98 


Avg. 


3.9 


4.0 


3.9 


3.9 


3.9 


Avg. 


3.9 


3.9 


3.9 


3.9 




3.8 


43. Usually work hard 
















44. Variety 




teaching methods 
























Mean All 


2-Yr 


BA,BS 


MA,MS Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 0 


0 0 




0 


0 


1.5 


0 


0 


0 


0 




0 
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2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.8 


0 


0 


0 


0 


0 


3.0 


1 


2 


1 


1 


1 


3.3 


12 


17 


9 


11 


11 


3.5 


32 


39 


26 


30 


31 


3.7 


57 


63 


51 


58 


56 


3.9 


80 


83 


77 


82 


79 


4.1 


92 


93 


91 


94 


92 


4.3 


97 


97 


97 


98 


98 


4.5 


99 


99 


99 


99 


99 


4.7 


99 


99 


99 


99 


99 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 3.6 3.6 3.7 3.6 

45. Students given responsibility 
achievement standards 


3.6 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.8 


0 


0 


0 


0 


0 


3.0 


0 


0 


0 


0 


0 


3.3 


0 


0 


0 


0 


0 


3.5 


1 


1 


1 


1 


2 


3.7 


4 


4 


4 


3 


5 


3.9 


11 


11 


11 


10 


14 


4.1 


25 


24 


25 


24 


28 


4.3 


46 


45 


47 


47 


50 


4.5 


71 


71 


72 


72 


73 


4.7 


89 


89 


90 


90 


90 


4.9 


98 


98 


98 


98 


98 


5.0 


99 


99 


99 


99 


99 


Avg. 


4.3 


4.3 


4.3 


4.3 


4.3 



47. Used educational technology 

Student Interest (4 items) 

Mean All 2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


2 


1 


4 


3 


2 


2.5 


9 


6 


13 


11 


9 


2.8 


16 


12 


22 


18 


15 


3.0 


21 


18 


29 


25 


21 


3.3 


32 


28 


41 


36 


31 


3.5 


40 


37 


49 


44 


39 


3.7 


49 


46 


57 


53 


48 


3.9 


58 


56 


66 


62 


58 


4.1 


68 


66 


74 


70 


67 


4.3 


77 


76 


81 


78 


77 


4.5 


86 


86 


89 


86 


86 


4.7 


93 


93 


95 


93 


93 


4.9 


98 


98 


98 


98 


98 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.6 


3.7 


3.5 


3.6 


3.6 
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2.0 


0 


0 


1 


1 


1 


2.5 


3 


2 


2 


3 


3 


2.8 


6 


4 


5 


7 


6 


3.0 


9 


7 


8 


10 


10 


3.3 


17 


14 


16 


18 


18 


3.5 


24 


22 


23 


26 


26 


3.7 


35 


32 


34 


37 


37 


3.9 


48 


47 


47 


50 


51 


4.1 


63 


62 


62 


65 


65 


4.3 


77 


78 


77 


79 


78 


4.5 


89 


90 


89 


90 


89 


4.7 


96 


96 


96 


97 


96 


4.9 


99 


99 


99 


99 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


3.8 


3.9 


3.8 


3.8 


3.8 



Mean 


46. High 

All 2-Yr BA,BS MA,MS Doct. 


1.5 


0 


0 


0 


0 


0 


2.0 


0 


0 


0 


0 


0 


2.5 


0 


0 


0 


0 


0 


2.8 


0 


0 


0 


0 


0 


3.0 


1 


1 


1 


0 


1 


3.3 


3 


2 


4 


2 


4 


3.5 


7 


6 


9 


6 


9 


3.7 


14 


12 


16 


13 


18 


3.9 


27 


25 


29 


27 


32 


4.1 


44 


42 


47 


44 


49 


4.3 


64 


63 


66 


64 


67 


4.5 


81 


81 


81 


81 


83 


4.7 


92 


93 


92 


92 


93 


4.9 


98 


98 


98 


98 


99 


5.0 


99 


99 


99 


99 


99 


Avg. 


4.1 


4.1 


4.1 


4.1 


4.1 



Mean 


Stimulating 

All 2-Yr BA.BS MA,MS Doct. 


10.0 


0 


0 


1 


0 


1 


11.0 


2 


1 


2 


2 


2 


12.0 


4 


3 


5 


4 


5 


13.0 


8 


7 


9 


8 


11 


14.0 


16 


13 


17 


16 


19 


15.0 


28 


24 


30 


29 


31 


15.5 


36 


32 


38 


38 


39 


16.0 


45 


40 


48 


47 


48 


16.5 


55 


51 


59 


58 


57 


17.0 


65 


61 


69 


68 


66 


17.5 


75 


72 


79 


78 


75 


18.0 


84 


82 


87 


86 


83 


18.5 


91 


90 


93 


92 


90 


19.0 


96 


95 


97 


97 


95 


20.0 


99 


99 


99 


99 


99 


Avg. 


15.9 


16.2 


15.8 


15.9 


15.8 
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Fostering Student Collaboration (3 items) Establishing 



Rapport (4 items) 



Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


Mean 


All 


2-Yr 


BA,BS 


MA,MS 


Doct. 


5.0 


0 


0 


0 


0 


0 


10.0 


0 


0 


1 


0 


1 


6.0 


1 


1 


1 


2 


2 


11.0 


1 


1 


2 


1 


2 


7.0 


5 


4 


5 


5 


5 


12.0 


3 


3 


3 


3 


4 


8.0 


11 


10 


11 


12 


11 


13.0 


7 


6 


7 


7 


9 


9.0 


19 


19 


20 


22 


18 


14.0 


14 


12 


14 


14 


16 


10.0 


30 


30 


32 


33 


28 


15.0 


25 


23 


25 


25 


27 


11.0 


44 


44 


47 


47 


41 


15.5 


32 


30 


34 


33 


35 


11.5 


52 


53 


56 


55 


48 


16.0 


41 


40 


43 


42 


43 


12.0 


61 


62 


66 


63 


58 


16.5 


51 


50 


54 


52 


53 


12.5 


70 


72 


76 


72 


67 


17.0 


62 


62 


65 


63 


63 


13.0 


79 


81 


84 


81 


76 


17.5 


73 


73 


76 


74 


72 


13.5 


88 


89 


92 


89 


85 


18.0 


83 


83 


86 


84 


82 


14.0 


94 


95 


96 


95 


93 


18.5 


91 


91 


92 


92 


90 


14.5 


98 


98 


99 


98 


97 


19.0 


96 


96 


97 


97 


95 


15.0 


99 


99 


99 


99 


99 


20.0 


99 


99 


99 


99 


99 


Avg. 


11.0 


11.0 


10.8 


10.9 


11.1 


Avg. 


16.1 


16.2 


16.0 


16.1 


16.0 



Encouraging Student Involvement (4 items) 

Classroom Experience (5 items) 

Mean All 2-Yr BA,BS MA,MS Doct. 


Mean 


All 


Structuring 

2-Yr BA,BS MA,MS Doct. 


10.0 


1 


1 


1 


1 


1 


13.0 


0 


0 


0 


0 


1 


11.0 


3 


3 


4 


4 


4 


15.0 


2 


1 


2 


2 


3 


12.0 


7 


6 


8 


8 


8 


17.0 


6 


5 


7 


7 


8 


13.0 


13 


12 


13 


14 


14 


18.0 


11 


8 


12 


11 


14 


14.0 


22 


20 


23 


23 


23 


19.0 


18 


14 


19 


19 


23 


15.0 


34 


32 


36 


36 


35 


20.0 


28 


23 


30 


30 


34 


15.5 


41 


40 


45 


44 


41 


20.5 


35 


29 


38 


37 


41 


16.0 


49 


48 


53 


53 


48 


21.0 


43 


37 


46 


45 


49 


16.5 


58 


58 


62 


61 


56 


21.5 


52 


45 


56 


54 


58 


17.0 


67 


68 


71 


70 


65 


22.0 


61 


55 


66 


64 


66 


17.5 


76 


77 


80 


78 


73 


22.5 


71 


66 


77 


75 


75 


18.0 


84 


86 


88 


86 


81 


23.0 


81 


77 


86 


84 


83 


18.5 


91 


92 


94 


92 


89 


23.5 


89 


87 


93 


91 


90 


19.0 


96 


96 


97 


96 


95 


24.0 


95 


94 


97 


96 


95 


20.0 


99 


99 


99 


99 


99 


25.0 


99 


99 


99 


99 


99 


Avg. 


15.6 


15.7 


15.5 


15.5 


15.7 


Avg. 


20.9 


21.3 


20.7 


20.8 


20.6 



Average ratings were generally about the same for institutions of various sizes (less than 1000; 1000-2499; 2500- 
4999; 5000-9999; and 10,000-1-). Of the 47 items, differences in average ratings among these groups exceeded 0.1 
on only 12. Results for these 12 items are shown in Table 10. 

Table 10 



^ypraap Ratlngg hv Tnig tltiitin nal Slz6 OH T^pIvp. Tt6?HS 



— — P 


All 

Classes 


Institutional Size 


<1,000 


1,000- 

2,499 


2,500- 

4,999 


5,000- 

9,999 


10,000 -1- 


5. Formed “teams” or “discussion 
groups” 


3.5 


3.3 


3.4 


3.6 


3.6 


3.5 
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1 1 . Related course to real life 
situations 


4.2 


4.1 


4.2 


4.2 


4.2 


4.3 


16. Asks students to share with diverse 
others 


3.7 


3.6 


3.6 


3.7 


3.7 


3.8 


17. Provided frequent feedback on 
tests 


4.1 


4.0 


4.0 


4.1 


4.1 


4.2 


20. Encouraged out-of-class 
interactions 


3.9 


3.7 


3.8 


3.9 


3.9 


3.9 


47. Used educational technology 


3.6 


3.5 


3.5 


3.6 


3.7 


3.7 


25. Progress on “team skills” 


3.5 


3.3 


3.3 


3.4 


3.5 


3.5 


26. Progress on “creative capacities” 


3.4 


3.5 


3.3 


3.4 


3.4 


3.4 


29. Progress on “finding, using 
resources” 


3.6 


3.5 


3.4 


3.5 


3.6 


3.6 


33. Amount of required reading 


3.2 


3.0 


3.1 


3.2 


3.2 


3.2 


35. Course difficulty 


3.4 


3.3 


3.4 


3.4 


3.5 


3.4 


36. Strong desire to take the course 


3.7 


3.7 


3.6 


3.6 


3.6 


3.8 



On most of these items, average ratings for institutions with the smallest enrollments tended to be lower than 
those for larger institutions. However, on an overall basis, the differences were too slight to conclude that 
institutional size had a significant influence on ratings. 



ERIC 



3G 



http 



://www.idea.ksu 



.edu/users/tchrptl2/TECHNICAL%20REPORT%2012.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 38 of 74 



II. The Structure of the Ratings 

Although students and faculty both rate 12 learning objectives, it is possible that a smaller number of 
“dimensions” would be adequate to describe “goals” or “progress.” Similarly, student ratings of 20 
teaching methods may well represent fewer than 20 teaching “styles.” 

To determine if there was a meaningful underlying structure to either the ratings of objectives or ratings 

[41 

of teaching methods, three Maximum Likelihood Factor Analyses with Orthogonal Rotation were 
conducted. One of these was for faculty ratings of the importance of the 12 objectives; a second was for 
student ratings of progress of these objectives; and the third was for student ratings of teaching methods. 
Results for both the Short and Diagnostic Forms were used in these analyses. 

In all analyses, factors with eigenvalues greater than 1.0 were extracted and rotated by the Varimax 
method. Rotated factor loadings of faculty ratings of the importance of the 12 objectives are shown in 
Table 11. 



Table 11 



Rotated Factor Loadings for 



Faculty Ratings of the Importance of 01 


jjectives 


Objective 


Factor I 


Factor 

II 


Factor 

III 


11. Learning to analyze and critically evaluate ideas, arguments, 
and points of view 


.71 


.09 


.02 


12. Acquiring an interest in learning more by asking questions and 
seeking answers 


.68 


.30 


.25 


8. Developing skill in expressing oneself orally or in writing 


.56 


.15 


-.31 


9. Learning how to find and use resources for answering questions 
or solving problems 


.54 


.42 


.12 


10. Developing a clearer understanding of, and commitment to, 
personal values 


.53 


.16 


.07 


7. Gaining a broader understanding and appreciation of 

intellectual/cultural activity (music, science, literature, etc.) 


.43 


-.04 


-.12 


6. Developing creative capacities (writing, inventing, designing, 
performing in art, music, drama, etc.) 


.35 


.33 


-.20 


4. Developing specific skills and points of view needed by 
professionals in the fields related to this course 


-.04 


.67 


.11 


5. Acquiring skills in working with others as a member of a team 


.33 


.43 


-.04 


3. Learning to apply course material (to improve thinking, 
problem solving, and decisions) 


.22 


.42 


.30 


2. Learning fundamental theories, principles 


.05 


.07 


.65 


1. Gaining factual knowledge (terminology, trends, etc) 


-.10 


.06 


.61 



Although the structure that emerged from this analysis was somewhat ambiguous, there were three 
relatively clear groupings of objectives. The first loading principally on Factor I, and included (in 
abbreviated form) Critical analysis, Interest in learning, Values development. Broad liberal education, 
and Communication skills. Taken together, these objectives seem to emphasize Intellectual 
Development. 

Three other objectives loaded primarily on Factor II — Professional skills, viewpoints; Applications; and 
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Team skills. The common focus of these objectives appears to be Professional Preparation. 

Finally, two objectives loaded primarily on Factor III — Principles and theories and Factual knowledge. 
These objectives both stress Basic Cognitive Development. 

The other two objectives (Creative capacities; Finding and using resources) appeared to represent a 
combination of Factor I (Intellectual Development) and Factor II (Professional Skills). Conceptually, 
then, faculty objectives centered on Basic Cognitive Development, a broader Intellectual Development, 
or Professional Preparation', but two objectives appeared to combine the last two of these. 

Did student ratings of their progress parallel faculty ratings of importance? Table 12 explores this 
question. 



Table 12 

Rotated Factor Loadings for 
Student Ratings of Progress on Objectives 



Objective 


Factor 

I 


Factor 

II 


8. Developing skill in expression myself orally or in writing 


.91 


.17 


6. Developing creative capacities 


.85 


.19 


11. Learning to analyze and critically evaluate ideas, arguments, and points of 
view 


.75 


.45 


10. Developing a clearer understanding of personal values 


.75 


.44 


7. Gaining a broader understanding and appreciation of intellectual7cultural 
activity (music, science, etc.) 


.73 


.26 


9. Learning how to find and use resources 


.62 


.53 


5. Acquiring skills in working as a member of a team 


.59 


.30 


2. Learning basic principles, generalization, or theories 


.22 


.92 


1. Learning factual knowledge (terminology, etc.) 


.18 


.91 


3. Learning to apply course material 


.44 


.79 


4. Developing professional competencies, points of view 


.43 


.78 


12. Acquiring an interest in learning more 


.63 


.66 



In this analysis, only two factors were extracted. The structure of progress ratings appears generally 
different from that of faculty “importance” ratings. The one clear similarity between the two involves 
the two objectives that had high loadings on Factor II but low ratings on Factor I in Table 12 (Principles 
and theories; Factual knowledge). This was called Basic Cognitive Development in the previous 
analysis, and might be labeled Building a Cognitive Background in the present analysis. 

All other objectives had substantial loadings on Factor I, ranging from .43 to .91, together with a wide 
range of loadings on Factor II. It can be inferred that all were perceived to involve cognitive 
development in addition to some other kind of development, represented by the Factor II rotated 
loading. An examination of the rotated loadings on both factors suggests that various combinations of 
fKacg io^d'”g£ represent differe”*^ ways stud'^nts their backg’"Oun(ls to advance educational 
competencies: 

1. Professional Development (Objectives 3 and 4; loadings on Factors I and II of .447.79 
and .437.78, respectively). 

2. Intellectual Development (Objectives 7, 10, and 11; loadings on Factors I and II 
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were .121.16, .757.44, and .757.45, respectively). 

3. Expressiveness (Objectives 6 and 8; loadings of .857.19 and .917.17). 

4. Life Long Learning Skills (Objectives 5, 9, and 12; loadings of .597.30, .627.53, and .637.66). 

Although the terminology suggested by the analysis of student ratings is similar to that used in 
describing faculty ratings, the two analyses do not always agree on the placement of individual 
objectives. They did agree that Basic Cognitive Development is being stressed by the first two 
objectives and that the third and fourth objectives related to Professional Development. Furthermore, 
Objectives 7, 10, and 1 1 were classified as Intellectual Development in both analyses. But 
Expressiveness and Life-Long Learning Skills, which seemed to emerge from the student analysis, were 
not evident as separate dimensions in the faculty ratings. 

It can be concluded that conceptualizations of faculty aspirations and student perceived outcomes have 
much in common. Both agree that conceptualization should include Basic Cognitive Development, 
Professional Development, and Intellectual Development. Student ratings offer two additional ways of 
conceptualizing the advancement of educational competencies — Expressiveness and Life Long Learning 
Skills. It should be noted that the two objectives not readily classified in the faculty analysis were 
included in the last two dimensions of the student analysis (Creative capacities as an Expressiveness 
objective and Finding, using resources as a Life Long Learning objective). 

It appears that the first two objectives are sufficiently redundant that, in subsequent revisions of the 
instrument, they could be combined. Other than that, the mathematical structures that emerged from 
these analyses were not very crisp. They may provide some guidance to those interested in developing 
conceptual schemes for describing the purposes of higher education, and will be used to classify the 
objectives in the IDEA Center’s Directions to Faculty. But they provided no reason to alter the current 
focus of the IDEA system on the relative importance of each individual objective. 

The final factor analysis was performed on student ratings of the 20 instructional methods Two factors 
were extracted. Rotated factor loadings are shown in Table 13. 
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Table 13 



Rotated Factor Loadings for Student Ratings ol 


Instructional Methods 


Method 


Factor I 


Factor II 


10. Explained material clearly and concisely 


.89 


.25 


6. Made it clear how each topic fit into course 


.86 


.35 


4. Demonstrated the importance of the subject matter 


.86 


.34 


12. Gave tests etc. that covered most important points 


.80 


.15 




13. Introduced stimulating ideas about the subject 


.78 


.48 


2. Found ways to help students answer own questions 


.76 


.51 


I. Displayed a personal interest in students 


.74 


.47 


3. Scheduled course work to help students stay up-to-date 


.74 


.36 


17. Provided timely and frequent feedback on tests etc. 


.69 


.28 


II. Related course material to real life situations 


.68 


.36 




8. Stimulated students to high intellectual effort 


.67 


.53 


7. Explained the reasons for criticisms 


.62 


.60 


20. Encouraged out-of-class student-faculty interaction 


.56 


.49 


15. Inspired students to set high achievement goals 


.60 


.69 




18. Asked students to help each other understand ideas 


.43 


.76 


16. Asked students to share ideas with diverse others 


.38 


.75 


19. Gave assessments that required original thinking 


.39 


.74 


9. Encouraged students to use multiple resources 


.35 


.66 




5. Formed “teams” or “discussion groups” 


.09 


.75 


14. Involved students in “hands on” experiences 


.27 


.75 



An examination of the rotated factor loadings suggests that the first factor focuses on the instructor’s 
role in transmitting knowledge while the second emphasizes the student’s role in acquiring knowledge. 

Within these broad categories, subgroups of items can be formed by attending to the relative size of the 
rotated loading on the two factors. The first subgroup (high loadings on Factor I; relatively low loadings 
on Factor II) appears to emphasize providing a clear classroom structure; the focus seems to be on 
course content. The next two item subgroups appear to center on increasing student motivation, a potent 
influence on learning. One aspect of motivation is reflected in the second subgroup (relatively high 
loadings on Factor I; moderate loadings on Factor II), which features ways of stimulating student 
interest. The four items in the next subgroup (where loadings on the two factors were nearly equal) 
emphasized a related approach to improving student motivation — methods designed to stimulate student 
effort. Although attracting interest in the subject is often the first step in motivating students, additional 
efforts may be required to encourage the student effort that learning requires. 

The final two subgroups both have high loadings on Factor II, the factor stressing the student’s role in 
learning. The first stresses involving students in learning activities, it reflects the adage that the best way 
to learn something is to teach it. The second emphasizes student interaction', activities requiring the 
exchange of student views or team participation represent another way instructors may facilitate 
learning. 
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Although the high inter-correlations among methods items resulted in a somewhat ambiguous factor 
structure, the sub-groupings of items make intuitive sense. Effective instruction requires attention to 
content; faculty members need to be not only authorities in their field but expert in organizing and 
communicating that content. Especially in lower division undergraduate courses, where student 
motivation is often low or marginal, the effective instructor must also attend to student readiness to 
learn, both by finding ways to capture student interest and by stimulating student effort. Although at 
times teaching is necessarily centered on the instructor’s input, effective instructors know that student 
learning is as much a function of what the student does as how the instructor proceeds. 

These “dimensions of effective teaching” are clearly not independent; a fact reflected in both the high 
item inter-correlations and the somewhat ambiguous factor structure. Classroom observations are 
consistent with this conclusion. Effective teachers typically organize and present class content. But at 
the same time, and sometimes with the same techniques, they elicit student interest, encourage student 
effort, and involve students in the teaching-learning process. It may be unwise and fruitless to 
conceptualize the “art” of teaching as a series of discrete and unrelated techniques. 

Prior to the conduct of these analyses, IDEA staff had proposed five a priori scales be developed using 
the 20 standard methods items. These scales were modeled after those developed by The National 

Survey of Student Engagement (NSSE)^ to describe features of the campus environment which 
promote student learning. Because the IDEA scales were limited to the classroom environment, and 
because they had not been empirically developed, they were given slightly different names than those 
employed by NSSE. They were called Stimulating Student Interest, Fostering Student Collaboration, 
Establishing Rapport, Encouraging Student Involvement, and Structuring the Classroom. The similarity 
of these names to those suggested for the five subgroups produced by the factor analysis is obvious, 
even though there was only a moderate overlap among the specific items included on “scales” with 
similar names. Although there would be a modest statistical advantage in revising the content of these 
scales in accordance with findings from the factor study, the advantages gained by refining the scales 
was judged to be outweighed by the disadvantage of sacrificing longitudinal comparisons. 

In summary, results from the factor analyses were relatively ambiguous. When methods were analyzed, 
five alternative approaches to instruction were identified. These approaches were far from independent, 
suggesting that the effective instructor must be prepared to adjust strategies to different times and 
circumstances. The analyses of objectives show that, while they could be grouped into a smaller number 
of categories, these groupings were not entirely distinct. Therefore, it seems advisable (with the possible 
exception of objectives concerned with basic cognitive development) to continue having instructors 
select the pattern of objectives that best describes their intentions without regard for how these 
objectives relate to each other. 
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III. The Process of Adjusting Ratings 

Teaching effectiveness is assessed in three ways — (1) the ratings of progress on individual objectives 
chosen as important or essential by the instructor; (2) the weighted average for objectives chosen by the 
instructor (Progress on Relevant Objectives - PRO); and (3) the three global measures (averages on As a 
result of taking this course, I have more positive feelings toward this field of study; Overall, I rate this 
instructor as an excellent teacher; and Overall, 1 rate this an excellent course. Effectiveness is reported 
in two ways — the simple average of student ratings on the measure and an “adjusted” measure. This 
section describes how “adjusted” scores were developed. 

Ratings are adjusted to take into account, insofar as possible, the fact that matters influence them that are 
beyond the instructor’s control. For example, if the majority of students were strongly motivated to take 
a class, ratings are likely to be higher than in classes with less interested students. Therefore, unless this 
is taken into account, instructors of highly motivated students would have an unfair advantage over 
those whose students were less interested and dedicated. 

In addition to size of class, the Diagnostic Form contains a number of items that are potentially relevant 
as measures of “extraneous circumstances.” The most apparent ones are Items 39 and 43 (/ really 
wanted to take this course regardless of who taught it; As a rule, I put forth more effort than other 
students on academic work.) For convenience, scores are these items are called “Course 
Motivation” (CM) and “Work Habits” (WH), respectively. 

Three other items were considered as relevant to potentially important extraneous circumstances — 
average ratings of Items 35, 36, and 37 (Difficulty of subject matter; I had a strong desire to take this 
course; and I worked harder on this course than on most courses I have taken). However, scores on 
these items could not be used as direct measures of extraneous influences because, at least in theory, 
each of them was, to a degree, under the control of the instructor. Obviously, the instructor controls 
many factors that make a course difficult or easy. Similarly, instructors can influence the amount of 
effort a student puts into a course. And, at least for some students, the desire to take a course may 
reflect the reputation its instructor has earned, a factor under the instructor’s control. 

Although ratings on these three items can be traced, in part, to instructor behavior or characteristics, they 
may also reflect factors that are not under the instructor’s control. Course difficulty may, for example, 
reflect the fact that disciplines differ on the degree to which they stress content that is inherently difficult 
(complex, obscure). Similarly, students may have a strong desire to take a course for reasons unrelated 
to the instructor’s reputation or behavior (the time of day the course was offered, the intent of friends to 
take the course, the need to satisfy some pre-requisite, etc.). And student effort may reflect, in addition 
to factors under the control of the instructor, such extraneous motivations as desire to be accepted in a 
professional school; desire to earn academic honors (or avoid academic dismissal); desire to impress 
someone else; etc. 

To determine whether ratings on any of these items represented extraneous influences that ought to be 
included in the adjustment process, an effort was made to exclude the portion of variation that could be 
accounted for by instructor behavior. The procedure was to conduct step-wise multiple regression 

r/'~\ 

analyses^ that employed each of these three measures as the dependent variable. For two of the items 
(difficulty and effort), 22 independent variables were employed (the 20 teaching methods items plus 
Items 33 and 34 — Amount of reading and Amount of other work. For Item 36 (/ had a strong desire to 
take this course), Item 38 (I really wanted to take a course from this instructor) was used as the 
independent variable. This permitted us to predict average ratings on each of these three items on the 
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basis of averages for the independent variables. 

This prediction represented the average rating expected on the basis of relevant student characteristics. 
By subtracting the prediction from the obtained average, we obtained a residual that represented the 
average on the item after the instructor’s influence had been removed. These residuals were labeled Dj^ 

(difficulty unrelated to the instructor), Ej^ (effort unrelated to the instructor), and OM (other 

motivation). A positive residual means that the average rating was higher than would be expected on the 
basis of the independent variablefs) . In other words, after the influence of the instructor’s approach to 
the class had been taken into account, student ratings of effort and difficulty were above average. The 
“difficulty” residual probably reflects differences among disciplines; some are inherently more 
challenging than others to the majority of students. The “effort” residual may reflect the adequacy of 
student background and/or student academic self-confidence. 

In initial analyses, 7 independent variables made significant contributions to the prediction of Item 35 
(difficulty); the same was true for Item 37 (Effort), although only 5 of the 7 significant variables were 
identical. In both instances, the partial regression weight for two of the measures was negative, a 
finding that invariably obscures interpretation. Furthermore, the amount of variance accounted for by 
two other measures was less than two percent of the total. 

In the interest of simplicity, new analyses were undertaken which employed only the three most 
important measures. For both difficulty and effort, these were the average ratings on Items 33 (amount 
of reading), 34 (amount of other work), and 8 (stimulating intellectual effort). The formula for 
predicting “difficulty” was: 

Predicted X35 = .13412 Xg + .23986 X33 + .40303 X34 + .74331; R^ = .371 
Dj^ = Mean of X33 - Predicted X33 

For “effort,” these formulas were: 

Predicted X37 = .35690 Xg + .11142 X33 + .51595 X34 +.06562; R^ = .635 
Ej^ = Mean of Xgy - Predicted Xgy 

Both formulas are easy to understand; the more reading is required, the more “other work” is required, 
and the more the instructor is perceived to stimulate intellectual effort, the more difficult the course is 
perceived to be and the more effort students report putting forth. and Ej^ tell us whether the 

difficulty and effort reported by students was more (positive residual) or less (negative residual) than 
was expected on the basis of instructor-controlled factors. 

Other motivation (OM) was calculated by predicting the mean for Item 36 (/ had a strong desire to take 
this course) from the mean of Item 38 (/ really wanted to take a course from this instructor) and 
subtracting the result from the obtained mean on Item 36. The formula was: 

Predicted Xgg = .57366 Xgg + i. 71732; R- = .527 
OM = Mean of Xgg - Predicted Xgg 

These results indicate that the desire to take a course can be partially explained by the desire to be 
exposed to a particular instructor. But a substantial portion of the variability in this measure is 
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apparently due to other (unspecified) motivations. 

The next step in the adjustment process was to conduct step-wise multiple regression analyses which 
employed the 12 ratings of progress and the 3 global ratings as dependent variables and six independent 
variables — enrollment (N), CM (mean of Item 39), WH (mean of Item 43), Dj,^, Ej,^, and OM. When this 

was done, the OM measure was statistically significant in only two analyses; and in these two, it 
contributed less than 1 percent to the explained variance. Therefore, this measure was dropped and 
analyses were repeated using only five independent variables. 

Table 14 provides information about statistically significant regression weights and other data needed to 
compute adjusted scores. Appendix B shows calculations for an example. 



Table 14 

Regression Coefficients and Constants for Adjusting Ratings On the Diagnostic Form 



Criterion 


Con- 

stant 


Regression Coefficient^ 


1+R^ 


Grand 

Mean 


CM 


WH 


N 


Dn 


En 


21. Factual 
knowledge 


1.69981 


.27568 


.38141 


— 


.09434 


-.07217 


1.176 


4.0013 


22. Principles 
and 

theories 


1.67498 


.25225 


.39835 


-.00065 


.09683 


-.12443 


1.163 


3.9443 


23. 

Applications 


1.55086 


.27966 


.43610 


-.00255 


-.10759 


-.12437 


1.225 


3.9874 


24. Prof skill, 
viewpoints 


1.45513 


.32015 


.42804 


-.00284 


-.09290 


-.06913 


1.238 


4.0420 


25. Team skills 


1.36271 


.20224 


.51612 


— 


-.26412 


-.11336 


1.161 


3.9285 


26. Creative 
Capacities 


1.74672 


.20146 


.45071 


-.01175 


-.47119 


.09341 


1.194 


3.8668 


27. Broad 
liberal 
education 


1.12469 


.24898 


.51462 


-.00463 


-.28984 


-.14497 


1.165 


3.6948 


28. 

Communication2 

skills 


17413 


.03283 


.44629 


-.00774 


-.57321 


— 


1.193 


3.7887 


29. Find, use 
resources 


1.34473 


.14364 


.54934 


-.00487 


-.19646 


-.17466 


1.169 


3.7322 


30. Values 

development 


1.15089 


.25370 


.47874 


— 


-.24761 


-.19709 


1.160 


3.7779 


31. Critical 
analysis 


1.96267 


.13407 


.42156 


-.00354 


-.19952 


-.15229 


1.119 


3.8438 


32. Interest in 
learning 


1.32320 


.26505 


.17280 


-.00578 


-.10333 


-.12346 


1.206 


3.7907 


40. Increased 
positive 
attitude 


1.00177 


.51242 


.33205 


-.00113 


-.22342 


.07431 


1.361 


3.8611 


41. Excellent 
teacher 


2.58021 


.24024 


.23139 


-.00122 


-.14747 


-.18191 


1.088 


4.1815 


42. Excellent 
course 


1.35036 


.47249 


.28732 


-.00136 


-.21410 


.05304 


1.294 


3.9198 



*CM=Course Motivation (item 39), WH=Work Habits (item 43), N=enrollment, Df^=Difficulty unrelated to the instructor, 



Ef,j=Effort unrelated to the instructor 

Note: Analyses reported in Table 14 are based on a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 
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It is clear from this table that “Work Habits” (WH, mean of Item 43) was generally the most potent 
predictor, followed by “Course Motivation” (CM, mean of Item 39). Classes that contained students 
who typically worked hard on their studies and/or were highly motivated to take the course regardless of 
who taught it were expected to receive favorable ratings; unless ratings were adjusted, the instructors of 
such classes would have an unfair advantage over colleagues with less motivated and less dedicated 
students. 

The joint effect of these two variables is displayed In Table 15. Classes were sorted into 5 groups on the 
basis of average scores on Item 39 (course motivation). The “Low” group’s average was in the lowest 
10 percent of all averages; “Low Average” was in the next 20 percent; “Average” was in the middle 40 
percent; “High Average” in the next 20 percent; and “High” in the upper 10 percent. Then each of these 
groups was sorted into five similarly defined groups on the basis of their average response to Item 43 
(work habits). The resulting 5x5 matrix produced 25 groups. Average progress ratings on each of the 
12 learning objectives for these 25 groups are shown in the table. The only classes included in this table 
were those for which the instructor identified the objective as “important” or “essential.” 

As seen in Table 15, the influence of these two variables on progress ratings is dramatized by comparing 
the two extreme groups (“Low/Low” vs. “High/High”). Differences ranged from 0.62 (for 
Communication Skills) to 1.17 (for Professional skills and viewpoints), averaging 0.96. Clearly, 
instructors in “High/High” classes have an enormous advantage over those in “Low/Low” classes; 
adjusted scores attempt to compensate for this advantage. 
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Table 15 

Average Progress Ratings for Classes That Differ in Levels of 
Student Motivation (Item 39) and Student Work Habits (Item 43) 



21. Gaining factual knowledge 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.51 


3.66 


3.80 


3.95 


4.08 


Low Avg. 


3.60 


3.76 


3.91 


4.05 


4.07 


Average 


3.73 


3.87 


4.02 


4.12 


4.21 


High Avg. 


3.88 


3.97 


4.13 


4.23 


4.33 


High 


4.01 


4.12 


4.25 


4.33 


4.48 


22. Principles^ 


theories 


Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.46 


3.64 


3.77 


3.89 


3.96 


Low Avg. 


3.58 


3.71 


3.86 


3.98 


3.98 


Average 


3.69 


3.83 


3.96 


4.05 


4.11 


High Avg. 


3.91 


3.94 


4.09 


4.15 


4.25 


High 


3.95 


4.10 


4.18 


4.26 


4.43 



23. Applications 



Work Habits 
(Item 43)' 


Student Motivation (Item 39) 


Low 


Low 

Avg^ 


Avg. 


High 

Avg. 


High 


Low 


3.53 


3.67 


3.75 


3.88 


3.96 


Low Avg. 


3.63 


3.73 


3.90 


4.00 


4.06 


Average 


3.69 


3.84 


4.00 


4.10 


4.23 


High Avg. 


3.85 


4.00 


4.12 


4.25 


4.34 


High 


3.98 


4.13 


4.25 


4.35 


4.53 


24, Professional skil 


viewpoints 


Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.38 


3.58 


3.78 


3.96 


4.11 


Low Avg. 


3.51 


3.70 


3.88 


4.05 


4.15 


Average 


3.64 


3.83 


4.01 


4.14 


4.28 


High Avg. 


3.76 


3.96 


4.14 


4.29 


4.38 


High 


4.04 


4.13 


4.28 


4.38 


4.55 



25. Team skills 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.49 


3.58 


3.66 


3.74 


3.75 


Low Avg. 


3.65 


3.68 


3.75 


3.86 


3.92 


Average 


3.67 


3.83 


3.92 


3.94 


4.09 


High Avg. 


3.81 


4.01 


4.06 


4.11 


4.16 


High 


3.94 


4.16 


4.26 


4.27 


4.47 






Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.46 


3.51 


3.54 


3.71 


3.85 


Low Avg. 


3.55 


3.61 


3.68 


3.87 


4.05 


Average 


3.57 


3.68 


3.83 


3.93 


4.12 
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High Avg. 


3.70 


3.88 


3.97 


4.08 


4.17 


High 


4.31 


4.03 


4.17 


4.26 


4.39 



27. Broad liberal education 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.15 


3.38 


3.45 


3.63 


3.81 


Low Avg. 


3.27 


3.50 


3.57 


3.68 


3.88 


Average 


3.42 


3.56 


3.74 


3.80 


3.99 


High Avg. 


3.44 


3.74 


3.86 


4.00 


3.97 


High 


3.75 


3.98 


4.04 


4.23 


4.28 



28. Communication skills 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.54 


3.63 


3.60 


3.57 


3.66 


Low Avg. 


3.64 


3.68 


3.67 


3.76 


3.71 


Average 


3.67 


3.76 


3.80 


3.79 


3.80 


High Avg. 


3.69 


3.91 


3.94 


3.91 


3.91 


High 


3.83 


4.01 


4.07 


4.08 


4.16 



29. Finding and using resources 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.45 


3.44 


3.49 


3.55 


3.65 


Low Avg. 


3.49 


3.56 


3.58 


3.65 


3.63 


Average 


3.57 


3.63 


3.71 


3.77 


3.85 


High Avg. 


3.63 


3.82 


3.87 


3.91 


3.99 


High 


3.86 


3.98 


4.08 


4.12 


4.27 



30. Values development 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.23 


3.42 


3.59 


3.71 


3.74 


Low Avg. 


3.41 


3.61 


3.66 


3.83 


3.87 


Average 


3.47 


3.64 


3.80 


3.85 


3.85 


High Avg. 


3.70 


3.81 


3.95 


4.03 


4.05 


High 


3.82 


3.91 


4.11 


4.17 


4.34 



31. Critical analysis 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.52 


3.62 


3.66 


3.80 


3.73 


Low Avg. 


3.60 


3.70 


3.75 


3.86 


3.83 


Average 


3.68 


3.78 


3.87 


3.89 


3.91 


High Avg. 


3.79 


3.92 


3.99 


4.02 


4.07 


High 


3.77 


4.02 


4.12 


4.17 


4.28 



32. Interest in continued learning 



Work Habits 
(Item 43) 


Student Motivation (Item 39) 


Low 


Low 

Avg. 


Avg. 


High 

Avg. 


High 


Low 


3.29 


3.45 


3.55 


3.71 


3.77 


Low Avg. 


3.41 


3.56 


3.65 


3.79 


3.93 


Average 


3.48 


3.63 


3.81 


3.89 


4.02 


High Avg. 


3.64 


3.82 


3.93 


4.02 


4.14 


High 


3.77 


4.00 


4.10 


4.19 


4.38 



ERJC 
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The regression coefficient for “Enrollment” (N) was not always statistically significant; but when it was, 
it was always negative, meaning the larger the class, the lower the predicted (expected) rating. Those 
teaching small classes have an advantage over those teaching large classes; hence, in the interest of 
fairness, ratings should be adjusted to take this into account. 

Except for the first two criterion ratings, the regression coefficient for was always negative. 

Generally, if the discipline was perceived as difficult (after taking into account the impact of the 
instructor on perceived difficulty), an attenuated outcome can be expected. This was especially apparent 
in progress ratings on “Creative capacities” and “Communication skills” where high difficulty was 
strongly associated with low progress ratings. The two exceptions, where “disciplinary difficulty” had a 
positive effect on the predicted outcome, were for the progress ratings concerned with basic cognitive 
development (“Factual knowledge” and “Principles and theories”). Consistent with other research 
regarding the influences of difficulty, this finding refutes conventional wisdom (high difficulty=low 
ratings). 

In most cases, student effort in the class (adjusted for the instructor’s influence on effort) was also 
negatively related to predicted ratings. Classes containing an unusually large number of students who 
worked harder than the instructor’s approach required ended up with lower progress ratings. As noted 
earlier, this may be because those who found it necessary to put in extra effort were those whose 
backgrounds did not prepare them well for the class. They may also be students who lack self- 
confidence and, for this reason, under-achieve (or under-estimate their progress in a self-abasing 
manner). 



Adjustments for the three global ratings merit special scrutiny. Regression results for predicted scores 
on “Increased positive attitude” and “Excellent course” were similar to each other. The order of the 
most influential predictors was reversed over that found for individual progress ratings; CM (desire to 
take the course regardless of who was teaching it) was the clear leader, and WH (tendency to work hard 
in academic studies) was a relatively distant second. Classes perceived as very difficult (D^^ were 

generally rated low on these measures, but (again in contrast to the findings for individual progress 
ratings) those with substantial numbers of students who worked hard in the class generally rated it more 
favorably. In other words, when students worked harder than required by the instructor, they tended to 
have good impressions of both the discipline and the course, even though their ratings of progress on 
relevant objectives tended to be low. But both global ratings and specific progress ratings tended to be 
low in disciplines perceived to be inherently difficult. 



The other global rating (“Excellent instructor”) was not predicted with much accuracy (R^=.0883); these 
measures of extraneous influences were not very predictive of students’ overall impressions of their 



instructors^. Although significant regression weights were found for all five independent variables, 
these were all of modest magnitude. CM and WH were about equal in their influence on such ratings, 
while the adjusted ratings for “Difficulty” and “Effort” had a more moderate (and negative) influence. 
Enrollment size had a very minor and negative influence. Thus, instructor “popularity” was not 



uccurciteiy predicLcd by llicsc iiicasuicS, biil sruucriL iTiOtiVatiOn and dcdiCatioii did haVC u iiicdcrcitc 
positive influence while disciplinary difficulty and student effort had a slight negative influence. 



The formula for adjusting means for progress ratings (Items 21-32) and global ratings (Items 40-42) is 
Grand Mean + (Obtained Mean - Predicted Mean)*(l + R^). This formula produces adjusted values 
with approximately the same mean and standard deviations as those obtained for unadjusted measures. 
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Adjustments to ratings on the Short Form were less precise because it provided no information on WH, 
Dj^ or Ej^. Since WH (work habits) was the most potent measure of relevant extraneous circumstances, 

its omission from the Short Form was especially regrettable. In later versions of this instrument, this 
item will be added. Until that time, it was decided to retain the adjustment formulas and process that 
have been in place since the 1998-99 school year. 

The formula for predicting OM (other motivation) was developed from Short Form results; it is similar 
to, but not identical with, that reported earlier for the Diagnostic Form. 

Predicted Mean of Item 13 = .519087 + 1.804711 

OM = Mean Item 13 - Predicted Mean, Item 13 



Table 16 provides information regarding regression coefficients and constants used in adjusting Short 
Form scores. 

Table 16 



Criterion 


Constant 


Regression Coe 


Ticient 


1+R^ 


Grand 

Mean 


CM 


OM 


N 


1. Factual knowledge 


2.83473 


.32094 


-.06596 


— 


1.102 


3.9038 


2. Principles and theories 


3.07102 


.23693 


— 


— 


1.084 


3.8526 


3. Applications 


2.87594 


.31386 


-.12552 


-.00239 


1.072 


3.8536 


4. Professional skills, viewpoints 


3.00560 


.30163 


— 


-.00262 


1.117 


3.9764 


5. Team skills 


1.92292 


.53771 


-.23726 


-.01384 


1.100 


3.3749 


6. Creative capacities 


3.18263 


.23181 


— 


-.00504 


1.070 


3.8348 


7. Broad liberal education 


3.12332 


.19650 


— 


-.00326 


1.034 


3.6707 


8. Communication skills 


3.57679 


.13616 


-.18760 


-.00951 


1.046 


3.8055 


9. Find, use resources 


2.42522 


.44526 


-.18993 


-.01693 


1.104 


3.4819 


10. Values development 


2.95472 


.26901 


-.14057 


-.00916 


1.090 


3.6285 


11. Critical analysis 


2.71324 


.27491 


-.10031 


-.00639 


1.072 


3.4837 


12. Interest in learning 


3.15930 


.16133 


-.15513 


— 


1.011 


3.7065 


16. Increased positive attitude 


2.28507 


.47865 


--- 


— 


1.212 


3.8708 


17. Excellent teacher 


2.63471 


.45726 


-.38354 


— 


1.060 


4.1496 


18. Excellent course 


2.22667 


.49763 


— 


— 


1.238 


3.8752 



Clearly, course motivation (CM) was the most important extraneous variable taken into account by 
adjustments to the Short Form; the stronger the desire of students to take the course regardless of who 
taught it, the more likely high progress ratings would be reported. The other two measures of influences 
beyond the instructor’s control (size of class and “other motivation”) did not always have significant 
regression weights. When they did, their weights were negative. If classes were large and/or if 
“extraneous” student motivation (motivation unrelated to a desire for a specific instructor) was low, it 
was probable that progress ratings would be negatively affected, making it necessary to adjust the 
ratings. 

To estimate the amount of improvement to Short Form adjustments which might be anticipated if the 
WH item were included, all calculations related to adjustments were performed using Diagnostic Form 
data but omitting Dj^ and Ej^, the measures of extraneous influences which would not be available on 

the Short Form. The amount of variance accounted for by extraneous measures (R^) increased from an 
average of .094 to an average of .156, a very substantial improvement (see Appendix C). 
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IV. Reliability 

Classes with 13-17 respondents were used to compute split half reliabilities for each of the 47 items and 
for the 5 teaching methods scales described in Section II of this report. Classes were randomly divided 
and means were computed for each half. These means were correlated. Results were taken as an 
estimate of the split half reliability of classes averaging 7.5 respondents. The Spearman-Brown 

Prophecy formula^ was applied to estimate reliabilities for classes averaging 12.5, 24.5, 42.5, and 60 
respondents (corresponding to class size ranges of 10-14, 15-34, 35-49, and 50-I-). 

191 

Standard deviations were also computed for each item^ or scale and these were used, in conjunction 
with the computed reliabilities, to calculate standard errors of estimate. Results are shown in Table 17. 

All measurements include a degree of “error.” The data of Table 17 provide the user with information 
about the likely range within which the “true” mean falls (the theoretical average from an infinite 
number of administrations of the form). In general, the probability that the true mean will fall within ± 
one standard error of the obtained mean is approximately two out of three; 95 times in 100 it will fall 
within two standard errors of the obtained mean. 
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Table 17 



Reliability and Standard Errors of Items and Scales 
For Four Class Sizes 



Teaching Methods 


All Classes 


Class Size 


10-14 


15-34 


35-49 


50+ 


Mean 


s.d. 


Hi 


s.e. 


Hi 


s.e. 


Hi 


S.e. 


Hi 


S.e. 


1. Displayed personal interest in 
students 


4.34 


.498 


.81 


.22. 


.89 


.17 


.93 


.13 


.95 


.11 


2. Helped students answer own 
questions 


4.10 


.520 


.79 


.24 


.88 


.18 


.93 


.14 


.95 


.12 


3. Scheduled work helpfully 


4.20 


.481 


.75 


.24 


.86 


.18 


.91 


.14 


.94 


.12 


4. Demonstrated imp of subject 


4.32 


.455 


.77 


.22 


.87 


.17 


.92 


.13 


.94 


.11 


5. Formed teams, discussion groups 


3.52 


1.03 


.90 


.33 


.95 


.24 


.97 


.18 


.98 


.16 


6. Made clear how topics fit 


4.20 


.506 


.77 


.24 


.87 


.18 


.92 


.14 


.94 


.12 


7. Explained criticisms 


3.78 


.570 


.72 


.30 


.84 


.23 


.90 


.18 


.93 


.16 


8. Stimulated intellectual effort 


3.86 


.573 


.77 


.27 


.87 


.21 


.92 


.17 


.94 


.14 


9. Encouraged use of multiple 
resources 


3.78 


.696 


.82 


.29 


.90 


.22 


.94 


.17 


.96 


.14 


10. Explained clearly 


4.12 


.610 


.83 


.25 


.91 


.19 


.94 


.15 


.96 


.12 


11. Related to real life 


4.22 


.581 


.82 


.25 


.90 


.19 


.94 


.14 


.96 


.12 


12. Tests covered important points 


4.28 


.492 


.79 


.23 


.88 


.17 


.93 


.13 


.95 


.11 


13. Introduced stimulating ideas 


4.03 


.583 


.81 


.25 


.89 


.19 


.94 


.15 


.95 


.13 


14. Involved students in hands on 
activities 


3.76 


.805 


.84 


.32 


.91 


.24 


.95 


.18 


.96 


.15 


15. Inspired students to set high 
goals 


3.76 


.621 


.78 


.29 


.88 


.22 


.92 


.17 


.95 


.15 


16. Asked students to share 
experiences 


3.69 


.790 


.84 


.32 


.91 


.24 


.95 


.19 


.96 


.16 


17. Provided timely feedback 


4.11 


.593 


.81 


.26 


.89 


.20 


.93 


.15 


.95 


.13 


18. Asked students to help each 
other 


3.79 


.642 


.79 


.30 


.88 


.22 


.93 


.17 


.95 


.15 


19. Assessments required creativity 


3.92 


.649 


.81 


.28 


.89 


.21 


.94 


.17 


.95 


.14 


20. Encouraged student/faculty 
contact 


3.90 


.627 


.78 


.29 


.88 


.22 


.92 


.17 


.95 


.15 


Learning Objectives 


21. Factual knowledge 


4.00 


.495 


.77 


.24 


.87 


.18 


.92 


.14 


.94 


.12 


22. Principles and theories 


3.94 


.485 


.76 


.24 


.86 


.18 


.91 


.14 


.94 


.12 


23. Applications 


3.99 


.516 


.75 


.26 


.85 


.20 


.91 


.16 


.93 


.13 


24. Professional skills, viewpoints 


4.04 


.424 


.75 


.21 


.86 


.16 


.91 


.13 


.94 


.11 


25. Team skills 


3.93 


.632 


.85 


.24 


.92 


.19 


.95 


.14 


.97 


.12 


26. Creative capacities 


3.87 


.701 


.83 


.29 


.91 


.21 


.95 


.16 


.96 


.14 


27. Broad liberal education 


3.69 


.731. 


.79 


.34 


.88 


.25 


.93 


.20 


.95 


.17 


28. Communication skills 


3.79 


.676 


.84 


.27 


.91 


.20 


.95 


.16 


.96 


.13 


29. Find, use resources 


3.73 


.571 


.75 


.28 


.86 


.22 


.91 


.17 


.94 


.14 


30. Values development 


3.78 


.629 


.79 


.29 


.88 


.22 


.93 


.17 


.95 


.14 


31. Critical analysis 


3.84 


.590 


.78 


.28 


.87 


.21 


.92 


.16 


.94 


.14 


32. Interest in learning 


3.79 


.562 


.73 


.29 


.84 


.22 


.90 


.18 


.93 


.15 


Course Ratings 


33. Amount of reading 


3.20 


.741 


.89 


.24 


.94 


.18 


.97 


.14 


.98 


.12 


34. Amount of other work 


■ 3.42 


.589 


.Si 


.26 


.oy 


1 n 
.ly 


r\A 


15 


95 


.13 


35. Difficulty of subject matter 


3.42 


.581 


.82 


.24 


.90 


.18 


.94 


.14 


.96 


.12 


Self-ratings 


36. Strong desire to take the course 


3.66 


.671 


.80 


.30 


.84 


.23 


.93 1 


.18 


.95 


.15 


37. Worked harder on this course 
than most 


3.57 


.557 


.77 


.27 


.87 


.20 


.92 


.16 


.94 


.14 
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38. Wanted this instructor 


3.40 


.675 


.80 


.30 


.89 


.23 


.93 


.18 


.95 


.15 


39. Wanted course regardless of 
instructor 


3.33 


.560 


.65 


.33 


.78 


.26 


.86 


.21 


.90 


.18 


43. Usually work hard on academic 
work 


3.64 


.308 


.39 


.24 


.56 


.20 


.69 


.17 


.76 


.15 


Global Ratings 


40. Increase positive attitude 
toward field 


3.86 


.602 


.75 


.30 


.86 


.23 


.91 


.18 


.94 


.15 


41. Excellent instructor 


4.18 


.643 


.83 


.26 


.91 


.20 


.94 


.15 


.96 


.13 


42. Excellent course 


3.92 


.607 


.80 


.27 


.89 


.21 


.93 


.16 


.95 


.14 


Progress on Relevant Objectives 
(PRO)“ 


50.9 


8.6 


.78 


4.0 


88 


3.0 


.92 


2.4 


.95 


2.0 



“pro ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation of 10. All other ratings 
were made on a 5-point scale where l=low and 5=high. 



Additional Method Items 


All Classes 


Class Size 


10-14 


15-34 


35-49 


5C 




Mean 


s.d. 


Hi 


s.e. 


Hi 


S.e. 


Hi 


s.e. 


Hi 


s.e. 


44. Used variety of evaluation 
methods 


3.83 


.596 


.75 


.30 


.85 


.23 


.91 


.18 


.94 


.15 


45. Expected students to take 
responsibility 


4.30 


.326 


.60 


.21 


.75 


.16 


.84 


.13 


.88 


.11 


46. High achievement standards 


4.12 


.413 


.69 


.23 


.81 


.18 


.88 


.14 


.91 


.12 


47. Used educational technology 


3.63 


.773 


.83 


.32 


.91 


.24 


.94 


.18 


.96 


.15 


Teaching Method 
Scales 


Stimulated Student Interest 


4.03 


.506 


.84 


.20 


.91 


.15 


.95 


.12 


.96 


.10 


Fostering Student Collaboration 


3.74 


.709 


.88 


.24 


.94 


.18 


.96 


.14 


.97 


.12 


Establishing Rapport 


4.06 


.490 


.83 


.20 


.91 


.15 


.95 


.12 


.96 


.10 


Encouraging Student Involvement 


3.97 


.560 


.86 


.21 


.92 


.16 


.95 


.12 


.97 


.10 


Structuring Classroom Experiences 


4.20 


.473 


.85 


.18 


.92 


.14 


.95 


.10 


.97 


.09 



Ratings were made on a 5-point scale where l=low and 5=high. 

Note: Analyses reported in Table 17 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 



For the five a priori scales, internal consistency reliabilities were computed using Cronbach’s Alpha. 

Since inter-correlations of items were generally high (see Table 6), these reliabilities were also 
high, as noted in Table 18. 



Table 18 



Internal Consistency Reliabi 


ities for Teaching Method Scales 


Scale 


Coefficient Alpha 


Stimulating Student Interest 


.935 


Fostering Student Collaboration 


.844 


Establishing Rapport 


.920 


Encouraging Student Involvement 


.852 


Structuring Classroom Experiences 


.928 



Note: Analyses reported in i able iS used a more resiricicd data acl. ClasscS with response rates less than 15 % cr not 
reporting the number enrolled were also excluded. 
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V. Validity 

What evidence is there that student ratings obtained from the IDEA system can be trusted? This 
section updates previous studies of the system’s validity based on results obtained in the most recent 
three years. Four approaches to validity were taken. 

1. The correlation of student progress ratings and instructor ratings of importance. 

The first study is based on three assumptions: (1) instruction is effective; (2) instructors make 
meaningful and conscientious judgments when they rate the importance of each objective; and (3) 
students make accurate ratings of the progress they make on these objectives (the validity question under 
investigation). If all three assumptions are true, then there should be a positive correlation between the 
instructor’s rating of importance and the students’ average rating of progress. To the degree that any of 
these assumptions is less than 100% true (instruction is not effective, instructors were not always 
conscientious in identifying objectives, students did not estimate their progress accurately) this 
correlation will be reduced. The correlation will also be attenuated by the fact that importance ratings 
are made using only a 3-point scale. For these reasons, this test of validity is considered to be a severe 
one. 

The bolded numbers in Table 5 provide the information required by this study. The average correlation 
between the instructor’s rating of importance and students’ average rating of progress on the 
corresponding objective across all 12 objectives was +.265. In contrast, the average correlation between 
instructor rating of importance of a given objective and student ratings of progress on the other 1 1 
(irrelevant) objectives was +.024. These findings are consistent with those reported for other samples 
dating back to 1973. We conclude that students rate their progress on instructional objectives with more 
than minimal validity. 

2. The consistency of student ratings with intuitive expectations. 

The 20 “methods” items included on the IDEA form were chosen because they have been identified as 
“desirable” or “potent” teaching techniques. Therefore, if student ratings are valid, there should be a 
degree of correspondence between their ratings of progress and their perceptions of how frequently the 
instructor employed these “potent” methods. The data of Table 6 make it apparent that the expected 
correspondence occurred almost uniformly. 

Aside from this expectation of general correspondence, there is the question of whether specific 
correlations make sense. An examination of relevant data in Table 6 shows that many intuitive 
expectations were met. For example, the teaching method most closely related to student ratings of 
progress on “Team skills” (Item 25) was Formed teams or "discussion groups” to facilitate learning 
(Item 5). Progress on “Learning to find and use resources for answering questions or solving 
problems” (Item 29) was most closely related to ratings of Encouraged students to use multiple 
resources to improve understanding (Item 9). Progress on “Developing a clearer understanding of, and 
commitment to, personal values” (Item 30) was most highly correlated with Asked students to share 
ideas and experiences with others whose backgrounds and viewpoints differ from their own (Item 16). 
Progress ratings on “Developing creative capacities” (Item 26) were most closely related to Gave 
projects, tests, or assignments that required original or creative thinking (Item 19). 

Data provided earlier with respect to the impact of class size on correlations between instructional 
methods and student progress provides additional evidence that student ratings were consistent with 
intuitive expectations (see Table 7). Progress ratings on “Developing creative capacities” (Item 26) 
were substantially related to Formed teams or "discussion groups” to facilitate learning (Item 5) for 
very large classes (where personalized techniques are more problematical), but not for smaller classes. 
And progress ratings on “Developing a clearer understanding of, and commitment to, personal 
values” (Item 30) was closely related to Asked students to help each other understand ideas and 
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concepts (Item 18) if class size was less than 35 but was not so useful in larger classes. 

3. The differential validity of the methods items. 

Teaching methods items that were most highly correlated with progress ratings were relatively 
distinctive for each objective (see Table 7). Exceptions were the first two objectives (basic cognitive 
background) and the third and fourth objectives (applications; professional skills and viewpoints) where 
identical lists of “most relevant” teaching techniques were identified. But when lists of the eight “most 
relevant” methods for “Factual knowledge” and “Team skills” were compared, only three were on 
both. Generally, with the exceptions noted above, the amount of overlap between any two sets of “most 
relevant” items was approximately 50 percent. Unless students were making differential judgments in 
answering the questions, such distinctive patterns of relevant teaching methods would not have existed. 

4. Correspondence between independently obtained student and faculty ratings. 

Using the Faculty Information Form (see Appendix A) faculty participants are asked to respond to a 
number of questions about the specific class they are teaching. Their answers to these questions 
sometimes suggest how students might rate their progress or otherwise evaluate the instructor and class. 
Several studies were undertaken to determine if these expected relationships existed. Their presence 
would constitute evidence for the validity of the system since the instructors and students each made 
their ratings without knowledge of each other’s views. 

In the first of these studies, instructors were asked to rate the impact of various circumstances on the 
learning of students (Contextual Question 4). Circumstances were described as having a “Positive,” “In 
between,” or “Negative” impact on learning. Four of them were believed to be especially relevant to 
overall (global) outcomes: previous experience in teaching the course; desire to teach the course; 
adequacy of students’ background and preparation for the course; and student enthusiasm. 

Table 19 compares the average rating on the four global criteria — progress on relevant objectives (PRO) 
and three single-item ratings (increased positive attitude toward the subject; excellent teacher; excellent 
course) — for classes that were rated as having different impacts on student learning. PRO results are 
reported in T Scores, while those for the three individual ratings are based on the IDEA system’s 5-point 
scale. 

In every instance, the expected differences were found. In classes where the circumstance was expected 
to have a positive influence on student learning, global ratings were significantly higher than in those 
where the expected impact was negative. Classes with “in between” faculty ratings invariably had “in 
between” student ratings on these four measures. 
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Table 19 

The Relationship Between Instructor Ratings of Selected Circumstances and Student Global 



Circumstance/ 
Expected Impact 


Global Rating 


PRO^ 


Increased Positive 
Attitude 


Excellent 

Teacher 


Excellent 

Course 


Previously taught 


Positive (N=19805) 


52.0 


3.93 


4.25 


3.99 


In between (N=2418) 


50.3 


3.81 


4.07 


3.81 


Negative (N=516) 


48.0 


3.66 


3.89 


3.62 


Desire to teach 


Positive (N=21333) 


51.9 


3.94 


4.24 


3.99 


In between (N=3228) 


49.4 


3.71 


4.01 


3.74 


Negative (N=192) 


48.7 


3.69 


3.97 


3.71 


Student background 




Positive (N=7164) 


52.8 


4.02 


4.27 


4.06 


In between (N=10386) 


51.7 


3.94 


4.24 


3.99 


Negative (N=5513) 


49.6 


3.69 


4.07 


3.75 


Student enthusiasm 




Positive (N=12214) 


52.8 


4.07 


4.31 


4.11 


In between (N=7514) 


51.2 


3.86 


4.18 


3.90 


Negative (N=3510) 


47.9 


3.50 


3.94 


3.56 



'pro (Progress on Relevant Objectives) ratings are standardized T Scores. The distribution has a mean of 50 and standard 
deviation of 10. All other ratings were made on a 5-point scale where l=low and 5=high. 

Note; Analyses reported in Table 19 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 



A second study focused on the instructor’s description of specific class emphases (Contextual Question 
3). They indicated whether the class required “None,” “Some,” or “Much” of seven activities: writing, 
oral communication, computer applications, group work, mathematical/quantitative work, critical 
thinking, and creative/artistic/design endeavor. If the IDEA system is valid (if both instructor and 
student ratings can be trusted), then there should be a relationship between some of these emphases and 
progress on related objectives. 



Specifically, if “writing” was emphasized, students should report above average progress on 
“Communication skills.” If “critical thinking” was emphasized, above average progress should be 
reported on “Critical analysis.” If “creative/artistic/design endeavor” was emphasized, students should 
report above average progress on “Creative capacities.” And if “group work” was emphasized, student 
progress on “Team skills” should be relatively high. 



Results are shown in Table 20. 
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Table 20 

Relationship Between Instructor Emphasis and 



Relevant Student Progress Ratings 



Student Progress Rating® 


Instructor Emphasis: Writing 




None 


Some 


Much 


Communication Skills 


Mean 


3.36 


3.61 


4.01 


S. D. 


.85 


.70 


.56 


N 


428 


5360 


6134 






Instructor Emphasis: Critical Thinking 




None 


Some 


Much 


Critical Analysis 


Mean 


3.54 


3.81 


4.07 


S.D 


.66 


.59 


.52 


N 


1005 


5777 


5131 






Instructor Emphasis: Creative Endeavor 




None 


Some 


Much 


Creative Capacities 


Mean 


3.52 


3.76 


3.99 


S. D. 


.83 


.74 


.61 


N 


959 


2561 


2606 






Instructor Emphasis: Groui 


Work 




None 


Some 


Much 


Team Skills 


Mean 


3.94 


3.99 


4.04 


S. D. 


.67 


.61 


.57 


N 


885 


4363 


3014 



“This study used only courses where the learning objective was selected as “important” or “essential,” making it a very 
conservative test of validity. 

Note: Analyses reported in Table 20 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 

All four F tests were highly significant (P<.0001). The expected relationships were confirmed, thus 
establishing validity for both instructor and student ratings. 

In a third validity test in which instructor and student ratings were compared the focus was on two 
objectives: Developing specific skills, competencies, and points of view needed by professionals in the 
field most closely related to this course and Gaining a broader understanding and appreciation of 
intellectual/cultural activity (music, science, literature, etc.). If the IDEA system is valid, the first of 
these should be chosen much more frequently by those teaching professionally oriented courses (or 
courses related to the students’ major field) while the second should be selected more frequently by 
instructors teaching courses directed to meeting general education or distribution requirements (as 
indicated by Contextual Question 5). 



This expectation was confirmed. More than 78 percent of those teaching professionally oriented courses 
chose the professional uevelopmeiir’ ubjcolivc, conipureu to 21 percent of those teaching general 
education/distribution courses. On the other hand, over 60 percent of the latter chose the “broad liberal 
education” objective compared to 39 percent of those teaching professionally oriented courses. 



Student progress ratings on these objectives were compared for the two types of classes; these 
comparisons were limited to classes for which the instructor chose the objective in question as 
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“relevant.” Results followed a similar pattern. Progress ratings were significantly higher on the 
professional development objective in professionally oriented courses (4.15 vs. 3.85 for classes focused 
on meeting general education/distribution requirements). Conversely, the latter averaged 3.72 on the 
broad liberal education objective compared to 3.63 for professionally oriented classes. In both 
instances, the “t” test was significant beyond the .001 level. 

Since both “relevance” and progress ratings were consistent with those expected if the IDEA system 
were valid, further confirmation of validity was provided. 

A final validity study centered on measures used to adjust student ratings. A number of studies have 
established that students give a much higher priority to courses that prepare them for a profession than 
for those aimed at a general or liberal education. Therefore, those teaching courses related to the 
student’s major interest should receive ratings indicative of higher student motivation than those 
teaching courses designed to meet general education or distribution requirements. Relevant measures of 
motivation are Items 36 and 39 (/ had a strong desire to take this course; I really wanted to take this 
course regardless of who was teaching it). Results of these two items for five types of classes are given 
in Table 21. Both F tests were significant beyond the .0001 level. 



Table 21 

Motivation Ratings by Principle Type of Student Enrolled in the Class 



Type of Student 


36. Strong desire to take 
tbis course 


39. Wanted to take course 
regardless of wbo taugbt it 


Mean 


s.d. 


Mean 


s.d. 


Lower Division, General 
Education 


3.34 


.65 


3.11 


.55 


Upper Division, General 
Education 


3.55 


.61 


3.21 


.54 


Lower Division, Specialized 


3.86 


.68 


3.49 


.55 


Upper Division, Specialized 


3.86 


.60 


3.44 


.51 


Graduate/Professional 


3.92 


.57 


3.49 


.49 



Ratings were made on a 5-point scale where l=low and 5=high 

Note: Analyses reported in Table 21 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 



The IDEA system makes adjustments in ratings to take this type of “extraneous circumstance” into 
account. If adjustments are successful in making the “playing field” more even, then they should be 
positive for those teaching general education courses and negative for those teaching courses related to 
the student’s major. Table 22 provides data to test the validity of this expectation (and hence the 
validity of adjustments). 

All F tests were significant (P<.0001). Without exception, adjustments for classes designed to meet 
general education/distribution requirements at the lower division level were positive, ranging from +.02 
to +.08 on individual objectives. At the upper division level, adjustments for this type of class were 
generally positive, although small negative figures were obtained on 4 of the 12 progress ratings. When 
pairwise comparisons were made, adjustments for upper division general education courses were 

Cl \aij. WVtl v»j. 

major/professional interests in 15 of the 16 comparisons. 

In most comparisons, adjustments for graduate/professional level courses were greater than those for the 
other four types. This was expected since students in such courses are almost always highly motivated. 

The high unadjusted ratings in these courses reflect, in part, this motivation^^^^. 
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Table 22 



Differences Between Adjusted and Unadjusted Ratings 
Among Five Types of Classes 



Criterion 


Type of Class 


General Education/ 
Distribution 


Specialized/Major 


Graduate/ 

Professional 


Louver 

Division 


Upper 

Division 


Lower 

Division 


Upper 

Division 


21. Factual knowledge 


+.08 


+.01 


-.06 


-.07 


-.06 


22. Principles and theories 


+.07 


+.01 


-.05 


-.07 


-.05 


23. Applications 


+.05 


.00 


-.04 


-.08 


-.11 


24. Professional skills, 
viewpoints 


+.05 


+.01 


-.03 


-.04 


-.08 


25. Team skills 


+.02 


-.02 


-.04 


-.08 


-.14 


26. Creative capacities 


+.06 


.00 


-.04 


-.10 


-.14 


27. Broad liberal education 


+.06 


-.01 


-.07 


-.12 


-.19 


28. Communication skills 


+.02 


-.03 


-.04 


-..04 


-.11 


29. Find, use resources 


+.06 


+.02 


-.02 


-.05 


-.08 


30. Values development 


+.06 


.00 


-.08 


-.07 


-.09 


31. Critical analysis 


+.02 


-.01 


-.04 


-.06 


-.09 


32. Interest in learning 


+.08 


+.02 


-.06 


-.09 


-.09 




Progress on Relevant 
Objectives^ 


+1.27 


+1.33 


-1.40 


-1.94 


-1.32 


Increased positive attitude 


+.08 


+.04 


-.10 


-.08 


-.11 


Excellent teacher 


+.04 


.00 


-.02 


-.05 


-.08 


Excellent course 


+.11 


+.06 


-.08 


-.08 


-.12 



“Progress on Relevant Objectives ratings are standardized T Scores. The distribution has a mean of 50 and standard deviation 
of 10. All other ratings were made on a 5-point scale where l=low and 5=high. 

Note: Analyses reported in Table 22 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 



Since these results were in line with expectations, it can be concluded that there is validity in the IDEA 
system’s adjustments. 



O 

ERIC 



58 



http://www.idea.ksu.edu/users/tchrptl2/TECHNICAL%20REP0RT%2012.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 61 of 74 



VI. Other Technical Questions 

This section addresses two questions that, while relevant to the interpretation of IDEA results, don’t fit 
into any of the previous five sections. These questions are: 

1. Are results on the Diagnostic and Short Form comparable? 

2. Are there significant differences among disciplines? 

1. Comparability of Diagnostic and Short Forms 

Initially, the two forms were compared by examining the averages for student ratings of progress on 
relevant objectives (those chosen as “Important” or “Essential” by the instructor) as well as on the three 
global ratings of effectiveness (increased positive attitude toward the subject, excellence of the teacher, 
and excellence of the course). Results are shown in Table 23. 



Table 23 

Comparison of Ratings on the IDEA Diagnostic Form 
And the IDEA Short Form 



Objective 


Diagnostic Form 




Short Form 


N 


Mean 


S. D. 


N 


Mean 


S. D. 


Factual knowledge 


31,990 


4.00 


.49 


21,301 


4.20 


.46 


Principles and theories 


30,394 


3.94 


.48 


20,404 


4.14 


.46 


Applications 


30,437 


3.99 


.52 


19,254 


4.12 


.49 


Professional skills, 
viewpoints 


21,564 


4.04 


.52 


15,042 


4.12 


.49 


Team skills 


12,085 


3.93 


.63 


7,307 


4.02 


.61 


Creative capacities 


9,288 


3.87 


.70 


7,419 


3.97 


.61 


Broad liberal education 


10,254 


3.69 


.73 


6,988 


3.89 


.65 


Communication skills 


18,170 


3.79 


.68 


10,944 


3.87 


.63 


Find, use resources 


15,652 


3.73 


.57 


9,690 


3.83 


.53 


Values development 


8,713 


3.78 


.63 


5,707 


3.87 


.60 


Critical analysis 


18,905 


3.84 


.59 


11,331 


3.96 


.55 


Interest in learning 


15,612 


3.79 


.56 


10,104 


3.92 


.53 


Overall Measure 


Increased positive attitude 


44,447 


3.86 


.60 




28,827 


3.98 


.58 


Excellent teacher 


44,447 


4.18 


.64 


28,827 


4.25 


.60 


Excellent course 


44,447 


3.92 


.61 


28,827 


4.00 


.59 



A consistent difference favoring the Short Form is apparent. For the 12 individual objectives, these 
differences averaged .119; for the three global ratings, they averaged .090. Differences of this 
magnitude are significant in both the statistical and the practical sense. The practicality of these 
differences is especially apparent when the distribution of ratings on the two forms is examined. See 



Table 24. 
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Table 24 

Diagnostic and Short Form Distribution of Means of 
Progress Ratings and Global Items (in Percentages) 



' 

Criterion 


Form“ 


Ran 


ge of Means 


<2.00 


2.00- 

2.49 


2.50- 

2.99 


3.00- 

3.49 


3.50- 

3.99 


4.00- 

4.49 


4.50+ 




D 


0.05 


0.34 


1.79 


8.04 


26.68 


42.28 


20.83 


21. Factual knowledge 


s 


0.01 


0.13 


0.78 


3.87 


16.81 


42.18 


36.21 




D 


0.04 


0.32 


2.11 


9.33 


28.78 


42.40 


16.01 


22. Principles and theories 


s 


0.02 


0.13 


0.95 


4.71 


20.11 


43.69 


30.39 




D 


0.05 


0.33 


2.15 


8.97 


26.62 


39.88 


22.00 


23. Applications 


s 


0.02 


0.21 


1.20 


5.73 


20.40 


41.32 


31.14 


24. Professional skills, 


D 


0.04 


0.36 


1.90 


8.08 


23.44 


39.18 


27.00 


viewpoints 


S 


0.03 


0.22 


1.21 


5.84 


20.51 


40.63 


31.56 




D 


0.29 


1.26 


3.72 


9.99 


23.25 


35.86 


25.63 


25. Team skills 


s 


0.09 


0.95 


3.43 


8.60 


20.54 


34.41 


31.98 




D 


0.59 


1.78 


5.25 


10.69 


22.89 


32.17 


26.64 


26. Creative capacities 


S 


0.21 


0.91 


3.11 


9.68 


22.88 


36.16 


27.04 




D 


0.75 


2.94 


7.88 


15.09 


24.68 


28.71 


19.95 


27. Broad liberal education 


S 


0.20 


1.54 


4.70 


12.69 


22.86 


32.68 


25.34 




D 


0.54 


1.85 


5.70 


13.23 


25.49 


33.37 


19.82 


28. Communication skills 


S 


0.26 


1.31 


4.52 


12.01 


25.36 


34.39 


22.13 




D 


0.15 


1.12 


5.56 


16.97 


32.91 


31.70 


11.60 


29. Find, use resources 


S 


0.02 


1.64 


3.56 


13.95 


32.96 


35.21 


13.67 




D 


0.30 


, 1.47 


5.61 


14.69 


28.12 


32.84 


16.98 


30. Values development 


S 


0.10 


0.96 


4.70 


13.31 


26.69 


32.91 


21.33 




D 


0.16 


1.09 


4.57 


12.51 


27.99 


36.53 


17.16 


31. Critical analysis 


S 


0.02 


0.58 


2.99 


10.74 


25.48 


37.25 


22.94 




D 


0.10 


0.87 


4.71 


15.17 


31.91 


33.52 


13.71 


32. Interest in learning 


S 


0.04 


0.42 


2.93 


10.88 


30.09 


37.12 


18.50 






D 


0.19 


1.00 


4.42 


12.57 


27.05 


34.59 


20.19 


40. Increased positive attitude 


S 


0.09 


0.70 


3.08 


9.46 


23.12 


36.77 


26.78 




D 


0.23 


0.82 


2.37 


5.88 


14.14 


28.79 


47.76 


41. Excellent teacher 


S 


0.13 


0.52 


1.86 


4.96 


13.24 


28.92 


50.38 




D 


0.16 


0.94 


3.79 


11.21 


24.94 


34.90 


24.06 


42. Excellent course 


S 


0.11 


0.67 


3.01 


9.10 


22.47 


35.70 


28.93 



‘*D=Diagnostic Form, S=Short Form 



A number of studies were conducted to try to account for these differences. 

One study restricted the comparison of the two forms to classes that were taught by the same method (e. 
g., “Lecture/Discussion,” “Skill/Activity,” etc.). No reduction in differences was found for these more 
homogeneous groups. 
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directed to the same audiences (lower division classes for students seeking to meet general education or 
distribution requirements; upper division classes directed to specialization interests of students; etc.). 
The advantage of Short Form users could not be accounted for by their tendency to teach different types 
of students than was true for Diagnostic Form users. 
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A special study was made of PRO and the three global ratings at eight institutions that had administered 
approximately equal numbers of both forms in at least 100 classes. Although in general the Short 
Form’s advantage was still apparent, there were some differences among institutions. Of the 32 
comparisons (4 measures for each of 8 institutions), the Short Form mean was higher in 20; but the 
Diagnostic Form had higher means 7 times, and the two were about equal on the other 5 comparisons. 

Disciplinary differences were examined by comparing results on the two forms for the eight disciplines 
where both forms were most commonly used. Differences were relatively small in Engineering and 
Communications departments, but relatively large in Philosophy and General Liberal Arts classes. This 
study was refined by restricting it to the 36 institutions that regularly employed both forms. “Within 
institutional disciplinary differences” were similar to those found when disciplinary differences were 
studied across all institutions. 

The most crucial test was made when the comparison was restricted to the 465 classes taught by the 
same instructor on two occasions — once using the Diagnostic Form and once using the Short Form. In 
this study, only 2 of the 15 comparisons produced significant differences; and the magnitude of the 
significant differences was about .10 less than that found in the original studies. 

Finally, the IDEA on-campus coordinators on campuses where substantial use was made of both forms 
were consulted. In most instances, these coordinators reported that the Short Form was employed with 
faculty members whose effectiveness had been well established (tenured faculty, others with significant 
amounts of experience, etc.). In contrast, the Diagnostic Form was typically required of junior, 
temporary, or part-time faculty. 

These reports offered strong support for the view that differences between the two forms were artifacts 
of campus policies that appeared to assure an advantage to the Short Form. When coupled with the 
findings for the “same course, same instructor” study, it was concluded that true differences between the 
two forms were, at most, minor. The decision to restrict all normative reporting to the Diagnostic Form 
meant that norms would reflect the full range of faculty users, not a set that represents established, 
veteran teachers. 

2. Disciplinary differences 

Do results on the IDEA forms differ for different disciplines? This question has been a major focus of 
IDEA’S research program. The short answer is, “Results differ significantly across disciplines, and 
some of these differences are substantial.” The question requires relatively complex and detailed 
analysis. Therefore, it will be addressed in the Center’s next technical report. In this report, a sample of 
disciplinary differences is provided below. 

A minimum of 500 classes was required before a discipline was considered in these analyses. A total of 
28 disciplines met this standard. Among other matters, the degree to which these disciplines identified 
each objective as “relevant” (“important” or “essential”) was determined. Similarly, for those classes in 
which the objective was chosen as relevant, the average progress rating was computed. These results are 
summarized below for two of the twelve objectives. Creative Capacities and Critical Analysis, in Table 
25. 
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Table 25 



Disciplinary Differences in Relevance and Progress Ratings 
For Two Learning Objectives 



Discipline 


Objective 


Creative ( 


[Rapacities 


Critical Analysis 


% 

Relevant^ 


Average 

Progress*’ 


% 

Relevant^ 


Average 

Progress*’ 


Accounting 


5.5 


3.06 


29.0 


3.64 


Admin/Management 


14.8 


3.66 


46.2 


3.98 


Art 


83.2 


4.38 


36.1 


3.78 


Biology/Life Science 


7.2 


3.15 


30.1 


3.61 


Business - General 


15.6 


3.65 


48.2 


3.83 


Chemistry 


5.8 


2.67 


26.7 


3.31 


Communications 


42.3 


4.13 


56.7 


3.98 


Computer/Information Sciences 


20.3 


3.46 


24.0 


3.37 


Design/Applied Arts 


69.0 


4.01 


40.4 


3.84 


Economics 


6.2 


2.82 


46.0 


3.65 


Education - General 


24.6 


4.06 


45.9 


4.07 


Engineering 


20.2 


3.31 


26.4 


3.38 


English Literature 


45.8 


4.27 


72.2 


4.10 


Fine and Applied Arts 


69.0 


4.17 


39.1 


3.83 


Foreign Language/Literature 


27.4 


3.71 


24.9 


3.65 


History 


17.6 


3.48 


69.3 


3.98 


Health Professions/Related 
Science 


8.8 


3.78 


32.5 


3.93 


Liberal Arts/General Studies 


29.0 


3.98 


67.6 


4.07 


Mathematics/Statistics 


6.3 


2.78 


22.8 


3.30 


Music 


64.1 


4.29 


19.6 


3.59 


Nursing 


7.7 


3.69 


42.0 


4.14 


Philosophy 


16.4 


3.64 


93.1 


4.37 


Physical Education/ Health/ Safety 


14.5 


3.60 


29.7 


3.63 


Physics 


6.7 


2.69 


36.1 


3.23 


Political Science/Govemment 


15.8 


3.47 


73.5 


4.17 


Psychology 


7.5 


3.54 


53.7 


3.93 


Religion 


13.7 


3.46 


60.1 


4.12 


Sociology 


13.9 


3.50 


64.9 


4.01 



“Percent identifying objective as “important” or “essential.” 

*^atings were made on a 5-point scale where l=low and 5=high. 

Note: Analyses reported in Table 25 used a more restricted data set. Classes with response rates less than 75% or not 
reporting the number enrolled were also excluded. 



Instructors indicated that gains in Creative Capacities represented an “Important” or “Essential” 
objective in over half of the classes in An, Design/' Applied Ails, Fine and Applied Arts, and fvxUStc. In 
contrast, it was considered “Of no more than minor importance” in over 90 percent of the classes in 
Accounting, Biological/Life Science, Chemistry, Economics, Health Professions, 
Mathematics/Statistics, Physics, and Psychology. The average progress rating in relevant (important; 
essential) classes was much higher in disciplines that featured this objective than in those where it was 
rarely chosen (4.21 for disciplines where this objective was popular; 3.13 for those where it was rarely 
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chosen). 

Findings for the Critical Analysis objective were similar. It was considered relevant in over two-thirds 
of the classes in English Literature, History, Liberal Arts/General Studies, and Philosophy (where it was 
rated as relevant in over 93 percent of all classes). But it was rated as relevant in fewer than twenty-five 
percent of the classes in Computer/Information Sciences, Foreign Language/Literature, 
Mathematics/Statistics, and Music. Again, progress ratings paralleled these differences, averaging 4.08 
for disciplines where it was commonly chosen and 3.48 for those where it was infrequently chosen. 

These findings illustrate some of the very large differences among disciplines. Because these are so 
extensive, a full accounting will be delayed until the publication of a subsequent technical report. 
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Appendix A 



Faculty Information Form 
Diagnostic Form 

Short Form (used Fall 1998-Summer 2002) 

Short Form (revised Fall 2002) 

Sample IDEA Report (Diagnostic Form) 

Sample IDEA Short Form Report (reflects adjustments described in Appendix C) 
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Appendix B 



Calculating Scores Reported in The IDEA Report 
(Diagnostic Form) for Individual Faculty Members 



Appendix A includes a sample of the report participants receive for each class. The figures on this 
report were computer-generated. For those who would like to calculate these figures by hand, either to 
check their accuracy or to get a better feel for what goes into a given calculation, Appendix B describes 
the process that is followed in making calculations. 
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I. Necessary Raw Data 

A. National data base results for progress ratings 
identified the objective as "Important” or "Essential”). 

Objective 

21. Gaining factual knowledge. . . 

22. Learning principles, theories. . . 

23. Applying course material. . . 

24. Developing professional skills, competency. . . 

25. Acquiring team skills. . . 

26. Developing creative capacities. . . 

27. Gaining a broad liberal education. . . 

28. Developing communication skills. . . 

29. Learning to find and use necessary recourses. . . 

30. Values development, clarification. . . 

31. Learning to critically evaluate. . . . 

32. Acquiring interest in learning more. . . 



(those in which the instructor 



Mean 


s. d. 


.R-_ 


4.0013 


.494 


.1761 


3.9443 


.485 


.1633 


3.9874 


.516 


.2248 


4.0420 


.524 


.2380 


3.9285 


.632 


.1611 


3.8668 .701 


.1940 




3.6948 


.732 


.1648 


3.7887 


.676 


.1930 


3.7322 


.571 


.1687 


3.7779 


.629 


.1599 


3.8438 .589 


.1186 




3.7907 


.561 


.2056 



on "relevant” classes 



B. Means and standard deviations of ratings on three "global outcomes” measures. 

Mean s. d. R- 

40. Increased positive feelings toward subject. . . 3.8611 .602 .3606 

41. Overall, instructor was excellent. . . 4.1815 .642 .0883 

42. Overall, course was excellent. . . 3.9198 .607 .2938 



C. Information from statistical detail (Section V, page 7, of IDEA Report) 



Progress on Essential Objectives 


Mean 

Reported 


Calculated 


21. Factual knowledge. . . 


3.7 


3.7241 


22. Principles, theories. . . 


3.7 


3.7241 


Progress on Important Obiectives 


23. Applying course material. . . 


4.1 


4.0690 


31. Learning to critically evaluate. . . 


3.9 3.9310 



Global Ratings 

40. Increased positive feelings toward subject. . . 

3.7 3.6552 

41. Overall, instructor was excellent. . . 4.6 4.5517 

42. Overall, course was excellent. . . 3.9 3.8966 



Items Needed to Make Adjustments 
39. Course motivation. . 



3.1034 

43. Work habits. . . 3.3 3.3448 

Number enrolled (page 1 of IDEA Report) 34 



8. Stimulated high intellectual effort. . . 

33. Amount of reading. . . 

34. Amount of other work. . . 



O 

ERIC 



3.8 



66 



3.1 

3.4 



3.7931 

3.1034 

3.3929 



http://www.idea.ksu.edu/users/tchrptl2/TECHNICAL%20REP0RT%2012.htm 



3.1 



10/15/02 



TECHNICAL REPORT 12 



Page 69 of 74 



35. Difficulty of course. . . 3.6 3.6429 

37. Effort (worked harder than normal. . .) 3.7 3.6897 



II. Preliminary Calculations 

A. Calculating Dj^ 

Dj^ = Mean of Item 35 minus Predicted Mean of Item 35; or Dj^ = - Pred 

Pred X 35 = .13412 Xg + .23986 X 33 + .40303 X 34 +.74331 (Technical Report , p. 37) 
= .5097 + .7436 + 1.3703 +.7433 = 3.3669 

Dj^ = 3.6429 - 3.3669 = .2760 

B. Calculating Ej^ 

Ej^ = X 3.7 - Pred X 3.7 

Pred X 3.7 = .35690 Xg + . 1 1 142 X 33 + .51595 X 34 + .06562 (Tech Report, p. 37) 

= 1.3562 + .3454 + 1.7542 + .0656 = 3.5214 
Ej^ = 3.6897 - 3.5214 = .1683 
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III. Calculating Adjusted Scores (from formulas on p. 38 of Technical Report) 

Adjusted progress rating on Item 21, Gaining factual knowledge. . . 

Predicted X 21 = .27568 X 39 + .38141 X 43 + .09434 -.0722 E^^ +1.69981 

= .8555 +1.2757 +.0260 -.0122 +1.6998 = 3.8448 

Residual = X 21 - Pred. X 2 j= 3.7241-3.8448 = -.1207 

Adjusted X 21 = Grand Mean, Item 21 +(Residual)(l + R^) 

= 4.0013 + (-1207)( 1.1761) = 4.0013 - .1420 = 3.8593 (IDEA Report, p. 3) 

Adjusted progress rating on Item 22, Learning principles and theories. . . 

Predicted X 22 = .25225X3g+.39835X43-.001N+.09683Dj,^-.1244Ej,^+1.67488 

= .7828 +1.3324 -.0340 +.0267 -.0209 +1.6750 = 3.7620 

Residual = X 22 - Pred X 22 = 3.7241 - 3.7620 = -.0379 

Adjusted X 22 = Grand Mean, Item 22 + (Residual)(l + R^) = 

3.9443 + (-.0379)( 1.1633) = 3.9443 - .0441 = 3.9002 (IDEA Report, p. 3) 

Adjusted progress rating on Item 23, Applications of course materials 

Predicted X 23 = .27966X39+.43610X43-.003N-.1076Dj,^-.1221Ej,^+1.055086 

= .8679 +1.4587 -.102 -.0297 -.0206 +1.5509 = 3.7252 

Residual = X 23 - Pred. X 23 = 4.0690 -3.7252 = .3438 

Adjusted X 23 = Grand Mean, Item 23 + (Residual)(l + R^) 

= 3.9874 + (.3438)( 1.2248) = 3.9874 + .4211 = 4.4085 (IDEA Report, p. 3) 

Adjusted progress rating on Item 31, Analysis and critical evaluation 

Predicted X 31 = .13407X39+.42156X43-.004N-.1995Dj,^-.1523Ej4+1.96267 

= .4160 +1.4100 -.136 -.0051 -.0256 +1.9627 = 3.5720 

Residual = X 31 - Pred. X 31 = 3.9310 - 3.5720 = .3590 

Adjusted X 3 J = Grand Mean, Item 31 + (Residual)(l + R^) 

= 3.8438 + (.3590)(1.1186) = 3.8438 + .4016 = 4.2454 (IDEA Report, p. 3) 

Adjusted rating. Item 40 — Increased positive attitude. 

Predicted X 40 = .51242X39+.33205X43-.001N-.2234Dj,^.+.0743Ej,^+1.00177 

= 1.5902 +1.1106 -.034 -.0617 +.0125 +1.0018 = 3.6194 

Residual = X 40 - Pred. X 4 Q = 3.6552 - 3.6194 = .0358 

Adjusted X 4 Q = Grand Mean, Item 40 + (Residual)(l + R^) 

= 3.8611 + (.0358)(1.3606) = 3.8611 + .0487 = 3.9098 (IDEA Report, p. 2) 



Adjusted rating. Item 41 — Excellence of teacher 

Predicted X 41 = .24024X39+.23139X43-.001N-.1475Dj,^.-.1819Ej4+2.58021 



= ./5zj -I-. / /40 -.034 -.0407 -.0306 

Residual = - Pred. = 4.5517 - 4.0012 = .5505 



I cor>o — A r>r>i o 



Adjusted X^q = Grand Mean, Item 41 + (Residual)(l + R^) 

= 4.1815 + (.5505)(1.0883) = 4.1815 + .5991 = 4.7806 (IDEA Report, p. 2) 



ErJc 68 

http://www.idea.ksu.edu/users/tchrptl2/TECHNICAL%20REP0RT%2012.htm 



10/15/02 



TECHNICAL REPORT 12 



Page 71 of 74 



Adjusted rating, Item 42 — Excellence of course. 

Predicted X 42 = .47249X39+.28732X43-.001N-.2141Dj,^.+.0530Ej,^+l. 35036 

= 1.4663 +0.9610 -.034 -.0591 +.0089 +1.3504 = 3.6935 

Residual = X 42 - Pred. X 42 = 3.8966 - 3.6935 = .2031 

Adjusted X 42 = Grand Mean, Item 42 + (Residual)(l + R^) 

= 3.9198 + (.2031X1.2938) = 3.9198 + .2628 = 4.1826 (IDEA Report, p. 2) 
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IV. Calculating T Scores 

T Score = 50+[10(Obtained Mean-Grand Mean) divided by s.d.], where Grand Mean is National Mean 
and s.d is National standard deviation. Obtained mean for unadjusted T Score is the raw mean. The 
Obtained mean for the adjusted T Score is the adjusted mean calculated above. 





Mean Scores 
A=Obtained-NatT 


B=Nat’l 
s. d. 


10(A/B) - IDEA Report Page 


Item 21 


Unadjusted 


3.7241-4.0013= -.2772 


.494 


-5.61 (+50 = 44, p. 3) 


Adjusted 


3.8593-4.0013= -.1420 


.494 


-2.87 (+50 = 47, p. 3) 


Item 22 


Unadjusted 


3.7241-3.9443= -.2202 


485 


-4.54 (+50 = 45, p. 3) 


Adjusted 


3.9002-3,9443= -.0441 


.485 


-0.91 (+50 = 49, p. 3) 


Item 23 


Unadjusted 


4.0690-3.9874= +.0816 


.516 


1.58 (+50 = 52, p. 3) 


Adjusted 


4.4085-3.9874= +.4211 


.516 


8.16 (+50 = 58, p. 3) 


Item 3 1 


Unadjusted 


3.9310-3.8438= +.0874 


.589 


1.48 (+50 = 51,p. 3) 


Adjusted 


4.2454-3.8438= +.4016 


.589 


6.82 (+50 = 57, p. 3) 


Item 40 


Unadjusted 


3.6552-3.8611= -.2059 


.602 


-3.42 (+50 = 47, p. 2) 


Adjusted 


3.9098-3.8611= +.0487 


.602 


0.81 (+50 = 51,p. 2) 


Item 41 


Unadjusted 


4.5517-4.1815= +.3702 


.642 


5.77 (+50 = 56, p. 2) 


Adjusted 


4.7806-4.1815= +.5991 


.642 


9.53 (+50 = 60, p. 2) 


Item 42 


Unadjusted 


3.8966-3.9198= -.0232 


.607 


0.38 (+50 = 50, p. 2) 


Adjusted 


4.1826-3.9198= +.2628 


.607 


4.53 (+50 = 55,p. 2) 



PRO 

Unadjusted 2(44 +45) + (52 +51) = 178 + 103 = 281 divided by 6 = 46.8 (p. 2) 
Adjusted 2(47 + 49) + (58 + 57) = 192+ 115 = 307 divided by 6 = 51.2 (p. 2) 



n 

I 
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Appendix C 



Regression Coefficients and Constants for 
Adjusting Ratings on the Revised Short Form 
Effective October 1, 2002 
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Adjusted 

Mean 


Constant 


CMi5 

Cl 


WHi 3 

C2 


#Enroll 

C3 


H-AdJ.R^ 


Grand 

Mean 


Item 21 


1.7559 


0.2572 


0.3842 


0 


1.1737 


4.0013 


Item 22 


1.7619 


0.2273 


0.3941 


0 


1.1593 


3.9443 


Item 23 


1.7019 


0.2663 


0.4096 


-0.00298 


1.2050 


3.9874 


Item 24 


1.5353 


0.3139 


0.4131 


-0.00303 


1.2304 


4.0420 


Item 25 


1.6622 


0.1700 


0.4742 


0 


1.1119 


3.9285 


Item 26 


1.8617 


0.2191 


0.4190 


-0.01188 


1.1201 


3.8668 


Item 27 


1.3038 


0.2344 


0.4871 


-0.00534 


1.1174 


3.6948 


Item 28 


2.4763 


0.0324 


0.3887 


-0.00849 


1.0599 


3.7887 


Item 29 


1.6477 


0.1114 


0.5054 


-0.00569 


1.1252 


3.7322 


Item 30 


1.4258 


0.2189 


0.4502 


0 


1.1088 


3.7779 


Item 31 


2.2063 


0.1118 


0.3839 


-0.00432 


1.0754 


3.8438 


Item 32 


1.4911 


0.2457 


0.4491 


-0.00624 


1.1881 


3.7907 


Item 40 


0.9700 


0.5363 


0.3222 


-0.00162 


1.3396 


3.8611 


Item 41 


2.8111 


0.2197 


0.1912 


-0.00182 


1.0600 


4.1815 


Item 42 


1.3442 


0.4922 


0.2748 


-0.00191 


1.2737 


3.9198 



CM, 5 =Course Motivation - Short Form Item 15. / really wanted to take this course regardless of who taught it. 
WH, 3 =Student Work Habits - Short Form Item 13. As a rule, I put forth more effort than other students on academic work. 

#Enrolled=Number of students enrolled in the course as indicated by the instructor on the Faculty Information Form. 

Note: Analyses are based on a more restricted data set. Classes with response rates less than 75% or not reporting the number 
enrolled were also excluded. 



^Copies of the instruments and sample copies of reports to participants are included in Appendix A. 

Institutions that were first-time participants in the IDEA program were excluded, as were classes with fewer than 10 
respondents. Furthermore, if a single institution contributed more than 5% of the classes processed in a given year, classes 
from that institution were randomly deleted until the remainder constituted only 5% of the total. 

*^T=50-i-[10(X-M)/SD] where X=mean for the instructor; M=mean for the comparison group; SD=standard deviation for 
the comparison group. 

^ Lawley, D. N. (1940) “The Estimation of Factor Loadings for the Method of Maximum Likelihood,” Proceedings/The 
Royal Society o/Edinburgh, 60, 64-82. Kaiser, H. F. (1958), “The Varimax Criterion for Analytic Rotation in Factor 
Analysis,” Psychometrika, 23, 187-200. 

National Survey of Student Engagement. National Benchmarks of Effective Educational Practice. Indiana University 
Center for Postsecondary Research and Planning: Bloomington, Indiana, 2001. 

^ Hocking, R. R. (1976) “The Analysis and Selection of Variables in Linear Regression,” Biometrics, 32, 1-50. 

^ Conceivably, this may be because ratings of this characteristic are determined almost exclusively by instructor behavior 
rather than by extraneous circumstances. Ratings on Item 10 Explained course material clearly and concisely, correlated .90 
with overall ratings of the instructor (Item 41). See Table 6. 



l + (n-l)rjj 

Standard deviations were calculated for the 44,447 classes with 10 or more respondents processed between 1998 and 

OAA1 A1 '3A n rvnc i c • frvr Anlv “r#».la.vant” f*.las.CP..Q Tthn.QP. for which the 

AI.W/A. .Lt.Wlllk3 t - j 

objective was selected as “important” or “essential”) were used in computing standard deviations. 

^^^Cronback, L. J. (1951) “Coefficient Alpha and the Internal Structure of Tests,” Psychometrika, 16, 297-334. 

Lower adjusted scores for such classes do not necessarily mean that unadjusted ratings overestimate instructional 
effectiveness. Rather, the quality of instruction is less vital in such classes since high student motivation and energy almost 
ensures high levels of progress. 
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